{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "51b552cd",
   "metadata": {},
   "source": [
    "## 第三章 神经网络\n",
    "\n",
    "+ 输入层\n",
    "\n",
    "+ 隐藏层\n",
    "\n",
    "+ 中间层\n",
    "\n",
    "### 激活函数（activation function）\n",
    "\n",
    "此激活函数以阈值为边界，一旦超出阈值，就切换输出，这样的函数也被称为“阶跃函数”。\n",
    "\n",
    "为什么要有激活函数呢？激活函数都是非线性函数。如果只有线性函数，那增加再多的神经网络层数也还是线性函数，加深神经网络就没有意义了。\n",
    "\n",
    "#### 阶跃函数\n",
    "$$\n",
    "h(x)\n",
    "\\begin{cases}\n",
    "0, &if\\ \\ x \\le 0\\\\[2ex]\n",
    "1, &if\\ \\ x \\gt 0\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "$$\n",
    "a = b + w_1 x_1 + w_2 x_2\n",
    "$$\n",
    "\n",
    "$$\n",
    "y = h(a)\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "33fd3fc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 阶跃函数\n",
    "\n",
    "def step_function_naive(x):\n",
    "    if x > 0:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "eb415d45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "print(step_function_naive(3.0)) # 1\n",
    "print(step_function_naive(-2.0)) # 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6a7fb528",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def step_function(x):\n",
    "    return np.array(x > 0, dtype=np.int32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7a18f156",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAiCklEQVR4nO3df1Bc9b3/8dcCZkk04DVpNsEQRKstlau5LjUFpTVW16I3Ux2n0rEjJkJHptFIqLaSzKjJOF+mrabUHxAzJqa20TL+bDtfqmG+c5tEo9+voXDbq+kPjbooIEJnWLQtuLvn+0eyS1YgsuTH+XA+z8cMM3JyDrzZ8SwvPud93sfnOI4jAAAAl2S4XQAAALAbYQQAALiKMAIAAFxFGAEAAK4ijAAAAFcRRgAAgKsIIwAAwFWEEQAA4KostwuYing8rp6eHs2dO1c+n8/tcgAAwBQ4jqPh4WHl5eUpI2Py9Y8ZEUZ6enqUn5/vdhkAAGAauru7tXjx4kn/fUaEkblz50o6+MPk5OS4XA0AAJiKSCSi/Pz85O/xycyIMJK4NJOTk0MYAQBghvmsFgsaWAEAgKsIIwAAwFWEEQAA4CrCCAAAcBVhBAAAuIowAgAAXEUYAQAAriKMAAAAVxFGAACAqwgjAADAVYQRAADgKsIIAABwFWEEAAC4ijACAABcRRgBAACuIowAAABXEUYAAICrCCMAAMBVhBEAAOAqwggAAHAVYQQAALiKMAIAAFxFGAEAAK5KO4zs3r1bK1asUF5ennw+n55//vnPPGbXrl0KBoPKzs7WmWeeqc2bN0+nVgAA4EFph5GPP/5Y559/vh566KEp7f/222/ryiuvVHl5uTo7O7Vu3TqtWbNGzzzzTNrFAgAA78lK94CKigpVVFRMef/NmzdryZIlampqkiQVFRVp3759uu+++3Tttdem++0BAIDHpB1G0vXKK68oFAqlbLviiiu0detWffLJJzrppJPGHTMyMqKRkZHk55FI5HiXCcAw//okpq0vva2Bj0Y+e2cAR+3aCxar+PRcV773cQ8jfX19CgQCKdsCgYCi0agGBga0aNGiccc0NjZqw4YNx7s0AAb7rz/36ycv/sXtMgBr/MeSf/NuGJEkn8+X8rnjOBNuT2hoaFB9fX3y80gkovz8/ONXIADjDP8rKkkqmDdH/3ne+D9aABxbZy84xbXvfdzDyMKFC9XX15eyrb+/X1lZWZo3b96Ex/j9fvn9/uNdGgCDxQ790XL2grm644ovulwNgOPpuM8ZKS0tVXt7e8q2nTt3qqSkZMJ+EQCQpGj8YBjJyph4BRWAd6QdRj766CN1dXWpq6tL0sFbd7u6uhQOhyUdvMRSVVWV3L+2tlbvvvuu6uvrtX//fm3btk1bt27V7bfffmx+AgCeFIvFJUmZmYQRwOvSvkyzb98+LV++PPl5orfjxhtv1Pbt29Xb25sMJpJUWFiotrY2rV27Vg8//LDy8vL0wAMPcFsvgCNKrIxkTtJbBsA70g4jl1xySbIBdSLbt28ft+1rX/ua/vCHP6T7rQBYLO5wmQawBc+mAWCk5MoIYQTwPMIIACPFYodWRugZATyPMALASKyMAPYgjAAwUowGVsAahBEARkoMPcvM4G0K8DrOcgBGSqyM0DMCeB9hBICRojF6RgBbEEYAGCkWPziBlTkjgPcRRgAYKXE3TQYNrIDnEUYAGIkJrIA9CCMAjJTsGaGBFfA8wggAIyXvpmFlBPA8wggAI9EzAtiDMALASDF6RgBrEEYAGCmW7BnhbQrwOs5yAEaK0jMCWIMwAsBIiaFnTGAFvI8wAsBIUZ7aC1iDMALASMmhZ8wZATyPMALASDwoD7AHYQSAkRh6BtiDMALASMmekQzepgCv4ywHYKRYMoy4XAiA447THICRYqyMANbgLAdgJHpGAHsQRgAYKcrQM8AahBEARhq7TEMYAbyOMALASImn9hJGAO8jjAAwUuKpvfSMAN5HGAFgpCiXaQBrEEYAGGnsbhrepgCv4ywHYKQoQ88Aa3CaAzBSnKFngDU4ywEYKcrQM8AahBEARmLOCGAPwggAIyUmsLIyAngfYQSAcRzH0aGFEWUQRgDPI4wAME7iEo3EyghgA8IIAONEDwsj9IwA3kcYAWCc1JUR3qYAr+MsB2Ccw1dGyCKA93GaAzBOnJURwCqc5QCMk7IyQssI4HmEEQDGiR02fdXnI40AXkcYAWCcxMAz7qQB7EAYAWAcRsEDdiGMADAOYQSwC2EEgHFiPLEXsAphBIBxosmVEd6iABtwpgMwDisjgF0IIwCME6VnBLDKtMJIc3OzCgsLlZ2drWAwqD179hxx/x07duj888/XnDlztGjRIq1atUqDg4PTKhiA99HACtgl7TDS2tqquro6rV+/Xp2dnSovL1dFRYXC4fCE+7/00kuqqqpSdXW1Xn/9dT311FN67bXXVFNTc9TFA/AmLtMAdkk7jGzatEnV1dWqqalRUVGRmpqalJ+fr5aWlgn3f/XVV3XGGWdozZo1Kiws1MUXX6ybb75Z+/btO+riAXgTQ88Au6QVRkZHR9XR0aFQKJSyPRQKae/evRMeU1ZWpvfee09tbW1yHEcffPCBnn76aV111VWTfp+RkRFFIpGUDwD24DINYJe0wsjAwIBisZgCgUDK9kAgoL6+vgmPKSsr044dO1RZWalZs2Zp4cKFOvXUU/Xggw9O+n0aGxuVm5ub/MjPz0+nTAAzHGEEsMu0Glg//eAqx3EmfZjVG2+8oTVr1uiuu+5SR0eHXnjhBb399tuqra2d9Os3NDRoaGgo+dHd3T2dMgHMUPSMAHbJSmfn+fPnKzMzc9wqSH9//7jVkoTGxkZddNFFuuOOOyRJ5513nk4++WSVl5fr3nvv1aJFi8Yd4/f75ff70ykNgIdway9gl7RWRmbNmqVgMKj29vaU7e3t7SorK5vwmH/84x/K+NQUxczMTEkHV1QA4NPGVkYYhQTYIO0zvb6+Xo8++qi2bdum/fv3a+3atQqHw8nLLg0NDaqqqkruv2LFCj377LNqaWnRgQMH9PLLL2vNmjW68MILlZeXd+x+EgCekVgZIYsAdkjrMo0kVVZWanBwUBs3blRvb6+Ki4vV1tamgoICSVJvb2/KzJGVK1dqeHhYDz30kL7//e/r1FNP1aWXXqof/ehHx+6nAOApcVZGAKv4nBlwrSQSiSg3N1dDQ0PKyclxuxwAx9nTHe/p9qf+W18753P6+U0Xul0OgGma6u9v/uwAYJzYoaFn3E0D2IEwAsA43E0D2IUwAsA4DD0D7EIYAWAcwghgF8IIAOMwgRWwC2EEgHHGekZ4iwJswJkOwDhjl2lcLgTACcGpDsA4MVZGAKtwpgMwTpSeEcAqhBEAxkkMPeNuGsAOhBEAxmFlBLALYQSAcWIx5owANiGMADBOzCGMADYhjAAwDkPPALsQRgAYh6FngF040wEYJ9EzkpXJyghgA8IIAOMkVkYyfIQRwAaEEQDGiTv0jAA2IYwAMM5YzwhhBLABYQSAcRITWOkZAexAGAFgnGiMnhHAJoQRAMahZwSwC2EEgHHoGQHsQhgBYJzkBFZ6RgArEEYAGCcaYwIrYBPOdADGSayMZNLACliBMALAODy1F7ALYQSAcaI8tRewCmEEgHESQ88yaWAFrEAYAWCcZAMrPSOAFQgjAIwT4zINYBXCCADj0MAK2IUwAsA4DD0D7EIYAWAchp4BduFMB2Achp4BdiGMADAOPSOAXQgjAIxDzwhgF8IIAONEY4eGnrEyAliBMALAOMwZAexCGAFgnMSzaTJoYAWsQBgBYJy4Q88IYBPCCADjJFZG6BkB7EAYAWCUeNzRoYURZTH0DLACZzoAoyRWRSSGngG2IIwAMEqiX0SSMukZAaxAGAFglMNXRri1F7ADYQSAUWKxw1ZGCCOAFQgjAIwSjceT/03PCGAHwggAoySmr/p8UgYrI4AVCCMAjJJ4Yi/9IoA9CCMAjBKNMfAMsM20wkhzc7MKCwuVnZ2tYDCoPXv2HHH/kZERrV+/XgUFBfL7/TrrrLO0bdu2aRUMwNvGHpLH30qALbLSPaC1tVV1dXVqbm7WRRddpEceeUQVFRV64403tGTJkgmPue666/TBBx9o69at+vznP6/+/n5Fo9GjLh6A9zAKHrBP2mFk06ZNqq6uVk1NjSSpqalJL774olpaWtTY2Dhu/xdeeEG7du3SgQMHdNppp0mSzjjjjKOrGoBnxQgjgHXSWgcdHR1VR0eHQqFQyvZQKKS9e/dOeMxvfvMblZSU6Mc//rFOP/10nXPOObr99tv1z3/+c9LvMzIyokgkkvIBwA6EEcA+aa2MDAwMKBaLKRAIpGwPBALq6+ub8JgDBw7opZdeUnZ2tp577jkNDAzoe9/7nv7+979P2jfS2NioDRs2pFMaAI8Y6xkhjAC2mFaHmO9Tg4gcxxm3LSEej8vn82nHjh268MILdeWVV2rTpk3avn37pKsjDQ0NGhoaSn50d3dPp0wAM1Bi6BkrI4A90loZmT9/vjIzM8etgvT3949bLUlYtGiRTj/9dOXm5ia3FRUVyXEcvffeezr77LPHHeP3++X3+9MpDYBHcJkGsE9aKyOzZs1SMBhUe3t7yvb29naVlZVNeMxFF12knp4effTRR8ltf/3rX5WRkaHFixdPo2QAXkYYAeyT9mWa+vp6Pfroo9q2bZv279+vtWvXKhwOq7a2VtLBSyxVVVXJ/a+//nrNmzdPq1at0htvvKHdu3frjjvu0E033aTZs2cfu58EgCfQMwLYJ+1beysrKzU4OKiNGzeqt7dXxcXFamtrU0FBgSSpt7dX4XA4uf8pp5yi9vZ23XrrrSopKdG8efN03XXX6d577z12PwUAzxibM8LQM8AWPsdxnM/ezV2RSES5ubkaGhpSTk6O2+UAOI7+68/9WrX9Nf376bn67a0Xu10OgKMw1d/f/OkBwCiJlRGe2AvYgzACwCj0jAD2IYwAMAp30wD2IYwAMEpi6BkrI4A9CCMAjMLKCGAfwggAo0QJI4B1CCMAjBKngRWwDmEEgFFYGQHsQxgBYJSxW3t5ewJswdkOwCgMPQPsQxgBYBR6RgD7EEYAGIWeEcA+hBEARokx9AywDmEEgFFYGQHsQxgBYBQmsAL2IYwAMAphBLAPYQSAUWLcTQNYhzACwChjPSO8PQG24GwHYBRWRgD7EEYAGCV66NZeJrAC9iCMADBK7GAWYWUEsAhhBIBREkPPuJsGsAdhBIBRovSMANYhjAAwCnNGAPsQRgAYhTAC2IcwAsAo3NoL2IcwAsAoDD0D7MPZDsAorIwA9iGMADBKYmWEoWeAPQgjAIwSZ2UEsA5hBIBRogw9A6xDGAFgFHpGAPsQRgAYJcqcEcA6hBEARmHoGWAfwggAoxBGAPsQRgAYZaxnhLcnwBac7QCMQs8IYB/CCACjcJkGsA9hBIBRCCOAfQgjAIzCnBHAPoQRAEZhAitgH8IIAKMkV0YyCSOALQgjAIySvJvGRxgBbEEYAWAUGlgB+xBGABiFoWeAfTjbARgleZmGnhHAGoQRAEbh1l7APoQRAMZwHCcZRjJoYAWsQRgBYIxDOUQSKyOATQgjAIyRGHgm0TMC2GRaYaS5uVmFhYXKzs5WMBjUnj17pnTcyy+/rKysLC1dunQ63xaAx8UOWxphZQSwR9phpLW1VXV1dVq/fr06OztVXl6uiooKhcPhIx43NDSkqqoqff3rX592sQC8LXpYGKFnBLBH2mFk06ZNqq6uVk1NjYqKitTU1KT8/Hy1tLQc8bibb75Z119/vUpLS6ddLABvi7MyAlgprTAyOjqqjo4OhUKhlO2hUEh79+6d9LjHHntMb731lu6+++4pfZ+RkRFFIpGUDwDed/jKCBNYAXukFUYGBgYUi8UUCARStgcCAfX19U14zN/+9jfdeeed2rFjh7Kysqb0fRobG5Wbm5v8yM/PT6dMADPU4aPgfVymAawxrQbWT79JOI4z4RtHLBbT9ddfrw0bNuicc86Z8tdvaGjQ0NBQ8qO7u3s6ZQKYYaI8lwaw0tSWKg6ZP3++MjMzx62C9Pf3j1stkaTh4WHt27dPnZ2duuWWWyRJ8XhcjuMoKytLO3fu1KWXXjruOL/fL7/fn05pADwgFuOJvYCN0loZmTVrloLBoNrb21O2t7e3q6ysbNz+OTk5+tOf/qSurq7kR21trb7whS+oq6tLy5YtO7rqAXhKzGEUPGCjtFZGJKm+vl433HCDSkpKVFpaqi1btigcDqu2tlbSwUss77//vh5//HFlZGSouLg45fgFCxYoOzt73HYAiB0aesbAM8AuaYeRyspKDQ4OauPGjert7VVxcbHa2tpUUFAgSert7f3MmSMAMJEoD8kDrORzHMf57N3cFYlElJubq6GhIeXk5LhdDoDj5H/eH9J/PviSAjl+/d91l7ldDoCjNNXf3zybBoAxkrf20sAKWIUwAsAYiQZWekYAuxBGABgjluwZ4a0JsAlnPABjRGMMPQNsRBgBYAx6RgA7EUYAGCPZM8LKCGAVwggAYySGnmXRwApYhTACwBj0jAB2IowAMEaMCayAlQgjAIyRGAefQQMrYBXCCABjxBNP7aVnBLAKYQSAMcZ6RnhrAmzCGQ/AGPSMAHYijAAwRqJnhLtpALsQRgAYIzFnhAmsgF0IIwCMkRwHTwMrYBXCCABjROkZAaxEGAFgjBg9I4CVCCMAjBHlqb2AlQgjAIwRjzP0DLARYQSAMbi1F7ATYQSAMcaGnvHWBNiEMx6AMVgZAexEGAFgjOTQM8IIYBXCCABjxA5mEcIIYBnCCABjJFZGGHoG2IUwAsAY9IwAdiKMADBGjHHwgJUIIwCMkVgZySCMAFYhjAAwRpyVEcBKhBEAxhjrGeGtCbAJZzwAY9AzAtiJMALAGNFDt/bSMwLYhTACwBiJoWesjAB2IYwAMAbj4AE7EUYAGCNKzwhgJcIIAGPEmMAKWIkwAsAYjIMH7EQYAWAMhp4BdiKMADAGQ88AO3HGAzAGQ88AOxFGABiDnhHAToQRAMZgzghgJ8IIAGNway9gJ8IIAGPQMwLYiTACwBj0jAB2IowAMAaXaQA7EUYAGIMwAtiJMALAGGM9I7w1ATbhjAdgDHpGADsRRgAYg7tpADtNK4w0NzersLBQ2dnZCgaD2rNnz6T7Pvvss7r88sv1uc99Tjk5OSotLdWLL7447YIBeFeUoWeAldIOI62traqrq9P69evV2dmp8vJyVVRUKBwOT7j/7t27dfnll6utrU0dHR1avny5VqxYoc7OzqMuHoC3HMoihBHAMj7HcZx0Dli2bJkuuOACtbS0JLcVFRXp6quvVmNj45S+xrnnnqvKykrdddddU9o/EokoNzdXQ0NDysnJSadcADPImQ3/W3FH+n/rvq4FOdlulwPgKE3193daKyOjo6Pq6OhQKBRK2R4KhbR3794pfY14PK7h4WGddtppk+4zMjKiSCSS8gHA2+JxR4daRlgZASyTVhgZGBhQLBZTIBBI2R4IBNTX1zelr3H//ffr448/1nXXXTfpPo2NjcrNzU1+5Ofnp1MmgBkodtgiLbf2AnaZ1hnv86X+1eI4zrhtE3nyySd1zz33qLW1VQsWLJh0v4aGBg0NDSU/uru7p1MmgBkkcSeNJJFFALtkpbPz/PnzlZmZOW4VpL+/f9xqyae1traqurpaTz31lC677LIj7uv3++X3+9MpDcAMd3gYYWUEsEtaZ/ysWbMUDAbV3t6esr29vV1lZWWTHvfkk09q5cqVeuKJJ3TVVVdNr1IAnhY9LIzQMwLYJa2VEUmqr6/XDTfcoJKSEpWWlmrLli0Kh8Oqra2VdPASy/vvv6/HH39c0sEgUlVVpZ/97Gf6yle+klxVmT17tnJzc4/hjwJgJktdGSGMADZJO4xUVlZqcHBQGzduVG9vr4qLi9XW1qaCggJJUm9vb8rMkUceeUTRaFSrV6/W6tWrk9tvvPFGbd++/eh/AgCekBh4JkkZhBHAKmnPGXEDc0YA7+sb+pe+0vh/lJXh05v/60q3ywFwDByXOSMAcLwwCh6wF2EEgBF4SB5gL8IIACMk7qZhZQSwD2EEgBFihBHAWoQRAEYYCyO8LQG24awHYAR6RgB7EUYAGIGeEcBehBEARogdurU3K5MwAtiGMALACNHYoZWRKTwBHIC3EEYAGCHmcJkGsBVhBIARuLUXsBdhBIAREg2s9IwA9iGMADBCjJ4RwFqEEQBGoGcEsBdhBIARxoae8bYE2IazHoARGHoG2IswAsAIDD0D7EUYAWCExNCzDBpYAesQRgAYIe7woDzAVoQRAEagZwSwF2EEgBFiDD0DrEUYAWCE5IPyuLUXsA5nPQAjJHpGWBgB7EMYAWCEsZ4R3pYA23DWAzDC2ARWlkYA2xBGABgh2TPCdRrAOoQRAEZITGDlqb2AfQgjAIzAU3sBexFGABghSs8IYC3CCAAjxOgZAaxFGAFgBFZGAHsRRgAYIXFrLw2sgH0IIwCMMNbAytsSYBvOegBGSPSM8KA8wD6EEQBGGBsHTxgBbEMYAWCExNAzGlgB+xBGABjh0FUaZdDACliHMALACMmVEXpGAOsQRgAYIfmgPC7TANYhjAAwQoyhZ4C1CCMAjJC4m4aeEcA+hBEARog7zBkBbEUYAWCEsZ4R3pYA23DWAzACPSOAvQgjAIwQPXRrL3fTAPYhjAAwAk/tBexFGAFghORTe2lgBaxDGAFghEQDKz0jgH0IIwCMEOOpvYC1CCMAjEDPCGCvaYWR5uZmFRYWKjs7W8FgUHv27Dni/rt27VIwGFR2drbOPPNMbd68eVrFAvCuGEPPAGulHUZaW1tVV1en9evXq7OzU+Xl5aqoqFA4HJ5w/7fffltXXnmlysvL1dnZqXXr1mnNmjV65plnjrp4AN7B0DPAXmmf9Zs2bVJ1dbVqampUVFSkpqYm5efnq6WlZcL9N2/erCVLlqipqUlFRUWqqanRTTfdpPvuu++oiwfgHQw9A+yVlc7Oo6Oj6ujo0J133pmyPRQKae/evRMe88orrygUCqVsu+KKK7R161Z98sknOumkk8YdMzIyopGRkeTnkUgknTKn7JmO9/Q/PUPH5WsDSM/QPz+RRAMrYKO0wsjAwIBisZgCgUDK9kAgoL6+vgmP6evrm3D/aDSqgYEBLVq0aNwxjY2N2rBhQzqlTcuuv36o3/x3z3H/PgCmLmf2+D9QAHhbWmEkwfepbnfHccZt+6z9J9qe0NDQoPr6+uTnkUhE+fn50yn1iC7/UkD5p80+5l8XwPScE5ir00/lnARsk1YYmT9/vjIzM8etgvT3949b/UhYuHDhhPtnZWVp3rx5Ex7j9/vl9/vTKW1aVpyfpxXn5x337wMAACaXVgPrrFmzFAwG1d7enrK9vb1dZWVlEx5TWlo6bv+dO3eqpKRkwn4RAABgl7Tvpqmvr9ejjz6qbdu2af/+/Vq7dq3C4bBqa2slHbzEUlVVldy/trZW7777rurr67V//35t27ZNW7du1e23337sfgoAADBjpd0zUllZqcHBQW3cuFG9vb0qLi5WW1ubCgoKJEm9vb0pM0cKCwvV1tamtWvX6uGHH1ZeXp4eeOABXXvttcfupwAAADOWz0l0kxosEokoNzdXQ0NDysnJcbscAAAwBVP9/c2oQwAA4CrCCAAAcBVhBAAAuIowAgAAXEUYAQAAriKMAAAAVxFGAACAqwgjAADAVYQRAADgKsIIAABwFWEEAAC4ijACAABcRRgBAACuIowAAABXEUYAAICrCCMAAMBVhBEAAOAqwggAAHAVYQQAALiKMAIAAFxFGAEAAK4ijAAAAFcRRgAAgKuy3C5gKhzHkSRFIhGXKwEAAFOV+L2d+D0+mRkRRoaHhyVJ+fn5LlcCAADSNTw8rNzc3En/3ed8VlwxQDweV09Pj+bOnSufz+d2Oa6LRCLKz89Xd3e3cnJy3C7H03itTxxe6xOH1/rEsf21dhxHw8PDysvLU0bG5J0hM2JlJCMjQ4sXL3a7DOPk5ORY+T+3G3itTxxe6xOH1/rEsfm1PtKKSAINrAAAwFWEEQAA4CrCyAzk9/t19913y+/3u12K5/Fanzi81icOr/WJw2s9NTOigRUAAHgXKyMAAMBVhBEAAOAqwggAAHAVYQQAALiKMOIRIyMjWrp0qXw+n7q6utwux3PeeecdVVdXq7CwULNnz9ZZZ52lu+++W6Ojo26X5hnNzc0qLCxUdna2gsGg9uzZ43ZJntPY2Kgvf/nLmjt3rhYsWKCrr75af/nLX9wuywqNjY3y+Xyqq6tzuxQjEUY84gc/+IHy8vLcLsOz/vznPysej+uRRx7R66+/rp/+9KfavHmz1q1b53ZpntDa2qq6ujqtX79enZ2dKi8vV0VFhcLhsNulecquXbu0evVqvfrqq2pvb1c0GlUoFNLHH3/sdmme9tprr2nLli0677zz3C7FWNza6wG/+93vVF9fr2eeeUbnnnuuOjs7tXTpUrfL8ryf/OQnamlp0YEDB9wuZcZbtmyZLrjgArW0tCS3FRUV6eqrr1ZjY6OLlXnbhx9+qAULFmjXrl366le/6nY5nvTRRx/pggsuUHNzs+69914tXbpUTU1NbpdlHFZGZrgPPvhA3/3ud/WLX/xCc+bMcbscqwwNDem0005zu4wZb3R0VB0dHQqFQinbQ6GQ9u7d61JVdhgaGpIk/j8+jlavXq2rrrpKl112mdulGG1GPCgPE3McRytXrlRtba1KSkr0zjvvuF2SNd566y09+OCDuv/++90uZcYbGBhQLBZTIBBI2R4IBNTX1+dSVd7nOI7q6+t18cUXq7i42O1yPOlXv/qVOjo6tG/fPrdLMR4rIwa655575PP5jvixb98+Pfjgg4pEImpoaHC75Blrqq/14Xp6evSNb3xD3/rWt1RTU+NS5d7j8/lSPnccZ9w2HDu33HKL/vjHP+rJJ590uxRP6u7u1m233aYdO3YoOzvb7XKMR8+IgQYGBjQwMHDEfc444wx9+9vf1m9/+9uUN+xYLKbMzEx95zvf0c9//vPjXeqMN9XXOvFm0tPTo+XLl2vZsmXavn27MjLI80drdHRUc+bM0VNPPaVrrrkmuf22225TV1eXdu3a5WJ13nTrrbfq+eef1+7du1VYWOh2OZ70/PPP65prrlFmZmZyWywWk8/nU0ZGhkZGRlL+zXaEkRksHA4rEokkP+/p6dEVV1yhp59+WsuWLdPixYtdrM573n//fS1fvlzBYFC//OUveSM5hpYtW6ZgMKjm5ubkti996Uv65je/SQPrMeQ4jm699VY999xz+v3vf6+zzz7b7ZI8a3h4WO+++27KtlWrVumLX/yifvjDH3Jp7FPoGZnBlixZkvL5KaecIkk666yzCCLHWE9Pjy655BItWbJE9913nz788MPkvy1cuNDFyryhvr5eN9xwg0pKSlRaWqotW7YoHA6rtrbW7dI8ZfXq1XriiSf061//WnPnzk325OTm5mr27NkuV+ctc+fOHRc4Tj75ZM2bN48gMgHCCDAFO3fu1Jtvvqk333xzXNBjcfHoVVZWanBwUBs3blRvb6+Ki4vV1tamgoICt0vzlMSt05dccknK9scee0wrV6488QUBh3CZBgAAuIruOwAA4CrCCAAAcBVhBAAAuIowAgAAXEUYAQAAriKMAAAAVxFGAACAqwgjAADAVYQRAADgKsIIAABwFWEEAAC4ijACAABc9f8BeER+uzdyLH0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pylab as plt\n",
    "\n",
    "X = np.arange(-5.0, 5.0, 0.1) # 步长0.1\n",
    "# print(X)\n",
    "Y = step_function(X)\n",
    "plt.plot(X, Y)\n",
    "plt.ylim(-0.1, 1.1)  # 指定图中绘制的y轴的范围\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96f9b0b4",
   "metadata": {},
   "source": [
    "### sigmoid函数\n",
    "\n",
    "阶跃函数，不连续，不可导，不平滑，不好！\n",
    "\n",
    "所以我们要找一个连续的，没有指数爆炸的函数\n",
    "\n",
    "+ **sigmoid**\n",
    "\n",
    "$$\n",
    "h(x) = \\frac {1}{1 + e^{-x}} \n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "86626f91",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "83b0b7af",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA5lklEQVR4nO3deVxVdeL/8fdlu+xXAUVQQNxRXKFIy2mnbLX6pk1Ntli/nKnMbJnMabO+w0zbONVoNW1TWdlqy9jCTJNaainhvq+ggAjIvewX7j2/Pyi+Q6IBAYd77+v5eNwHcTgH3tyH3PvunM/5fCyGYRgCAAAwiZ/ZAQAAgG+jjAAAAFNRRgAAgKkoIwAAwFSUEQAAYCrKCAAAMBVlBAAAmIoyAgAATBVgdoDWcLvdKigoUEREhCwWi9lxAABAKxiGoYqKCsXHx8vP79jnPzyijBQUFCghIcHsGAAAoB3y8/PVr1+/Y37dI8pIRESEpMZfJjIy0uQ0AACgNRwOhxISEprex4/FI8rIj5dmIiMjKSMAAHiYnxtiwQBWAABgKsoIAAAwFWUEAACYijICAABMRRkBAACmoowAAABTUUYAAICpKCMAAMBUlBEAAGAqyggAADAVZQQAAJiKMgIAAExFGQEAAKaijAAAAFNRRgAAgKkoIwAAwFSUEQAAYCrKCAAAMBVlBAAAmIoyAgAATEUZAQAApqKMAAAAU1FGAACAqdpcRpYvX64LL7xQ8fHxslgsWrJkyc8es2zZMqWlpSk4OFgDBgzQs88+256sAADAC7W5jFRVVWn06NF65plnWrX/3r17dd5552nixInKzc3Vvffeq5kzZ+q9995rc1gAAOB9Atp6wKRJkzRp0qRW7//ss88qMTFR8+fPlySlpKRo7dq1evzxx3XZZZe19ccDAAAv0+ljRlatWqXMzMxm28455xytXbtW9fX1LR5TV1cnh8PR7AEAALxTp5eRoqIixcbGNtsWGxurhoYGlZSUtHhMVlaWbDZb0yMhIaGzYwIAAJN0yd00Foul2eeGYbS4/Udz5syR3W5veuTn53d6RgAAvJ1hGKpxulRcUau9JVXadNCuVbtL9a8th3TIUWtarjaPGWmrPn36qKioqNm24uJiBQQEKDo6usVjrFarrFZrZ0cDAMDjGIYhR22DjlQ5VV5Tr/Jqp+w19SqvbnzYa+rlqK2Xo+ljgyrq6lVR26DK2gY1uI0Wv+/frhyn80fFdfFv06jTy8j48eP18ccfN9v2xRdfKD09XYGBgZ394wEA6PZq610qqazT4YofHpV1KqlwqrSqTqVVTpVW1qm00qkj1Y3l41iForUsFik8KEBh1gCFWf0Vbg1QqNW/g36btmtzGamsrNSuXbuaPt+7d6/WrVunqKgoJSYmas6cOTp48KBeffVVSdKMGTP0zDPPaPbs2brxxhu1atUqvfjii3rzzTc77rcAAKCbqqitV6G9VgXlNSq016rwh4+HKup0yF6rQxW1Kq9u+YaO4wkN8lfP0CD1CA1sfIQEKTIkULYfHpEhAYoMDlREcIAiggMV+cPH8OAAhQb6y8+v5aESZmhzGVm7dq1OP/30ps9nz54tSbrmmmv0yiuvqLCwUHl5eU1fT05O1tKlS3X77bfrb3/7m+Lj4/XUU09xWy8AwCvUNbiUX1ajvLIq7S+tVn5ZjQ4cqdaBI40fHbUNrfo+Qf5+6hVhbXrEhAcpJtyqqLAgRYdbFR0WpKiwoKYCEhxo3pmMjmYxfhxN2o05HA7ZbDbZ7XZFRkaaHQcA4GPcbkMHy2u0p6RKew5Xas/hKu0pqdS+kmoV2Gv0c++ktpBAxdmCFd8jRHG2YMXZghUb+d8Pq2whgce8scNTtfb9u9PHjAAA4CkMw1BxRZ22FDq0o6hCOw5VamdxhXYVV6ra6TrmcWFB/kqMDlNSVKgSo0PVr2fID49Q9e0RojArb7fHw7MDAPBJbrehfaVV2njQrk0H7dpS6NDWwgqVVTlb3D/Q36L+0WEa0CtMyTHhGtArTANiwpQUHaaY8CCvO6vRlSgjAACfUOyo1fd55crNO6L1B8q1+aBDFXVHj+fws0gDeoVrWJ8IDYmN0ODe4RocG6H+0aEK8Gex+85AGQEAeB2329DO4kp9u7dUa/Yd0ff7j+hgec1R+1kD/DQ8PlIj+9o0Ij5SKXGRGhIb4VWDQz0BZQQA4PEMw9COQ5X6eleJVu8p1Zp9ZUfdLmuxSENjIzQ2safGJNg0ql8PDe4dztmOboAyAgDwSIcctVq+47C+2VWir3eVqqSyrtnXQwL9lZbUUyf0j1J6/54a1c+miGAm2+yOKCMAAI/gchtaf6Bc/9lWrC+3FWtzQfMV3YMD/ZSRHK3xA6OVkRyl1L42BXLWwyNQRgAA3VZdg0srd5fq801Fyt5ySKX/daeLxSKN6mvTxMG9dPKgGI1L6iFrAGM9PBFlBADQrdTWu/TV9sP6dFOhvtxa3OyOl4jgAP1qSC+dPrS3ThvaSzHhLKrqDSgjAADTNbjcWrWnVB+tK9Bnm4qaFZDYSKsyh/fRual9dGJyFJdevBBlBABgmp2HKvROzgF9kHtQhyv+bwBqnC1YF4yK06SRcRrTr0e3WtQNHY8yAgDoUhW19fpkQ6HeXpuv3Lzypu09QwN13sg4XTymr9KTelJAfAhlBADQJbYVOfTaqv36IPdg0zov/n4WnTGsty5P66fTh/XmEoyPoowAADpNvcutTzcV6fVV+/XdvrKm7YN6h2tqeoImj+2rXhEMQvV1lBEAQIerqK3X4jX5eunrvSqw10pqPAty7og++s1JSTppQBQLy6EJZQQA0GGK7LV6eeVevbE6r+mOmJjwIF2VkaQrMxIVGxlsckJ0R5QRAMAvdrC8Rgu/2qW31xyQ0+WWJA3sFaYbJw7Q5LF9WXgOx0UZAQC0W35ZtRZ8tVvv5uSr3mVIkk7o31M3/WqgzhjWmzti0CqUEQBAmxU7avXXf+/U4jX5anA3lpDxA6J121mDddKAaJPTwdNQRgAArVZRW6/nl+/RCyv2qqa+8fbcUwbFaOaZg3VicpTJ6eCpKCMAgJ9V73LrtVX79fSXO3Wkul6SNDaxh+45d5gyOBOCX4gyAgA4rmU7Dmvex5u1+3CVJGlArzDdfc4wnTMilttz0SEoIwCAFu0rqdIj/9yif20tliRFhwXpjsyhmpLeTwHMlIoORBkBADRTW+/S3/6zS88t2yOny60AP4uumdBfM88cLFtIoNnx4IUoIwCAJit3l2juB5u0t6TxksyvhvTS/RekaFDvCJOTwZtRRgAAOlLl1B+XbtU7OQckSb0jrHroohE6N7UP40LQ6SgjAODjPttUpD8s2aiSSqcsFuk3GUm669yhigzmkgy6BmUEAHyUvaZeD320We/nHpQkDYkNV9alI5WWxHwh6FqUEQDwQSt2Htbd725Qob1WfhZpxqkDddtZg2UNYA0ZdD3KCAD4kNp6l7KWbtU/Vu2XJPWPDtUTU8YoLamnycngyygjAOAj9hyu1C1v5GpLoUOSNG18ku6ZNEyhQbwVwFz8CwQAH/D+9wf0hyWbVO10KSosSE9MGa3Th/Y2OxYgiTICAF6t2tmgPyzZpPe/bxykOn5AtOZfMUaxkcEmJwP+D2UEALzU/tIq3fRajrYVVcjPIs06a4huPn2Q/P2YNwTdC2UEALzQV9uLNfPNXDlqGxQTbtUzV47VSayui26KMgIAXsQwDC34arce/2K7DEMak9BDz/4mTX1sXJZB90UZAQAvUe1s0B1vr9enm4okSb8+MVEPXjScuUPQ7VFGAMALFDtqNf0fa7XxoF1B/n566OIR+vWJiWbHAlqFMgIAHm5roUPTX1mjAnutosKC9PzVaUrvz5Tu8ByUEQDwYP/ZVqxb3vheVU6XBvQK08vXnqCk6DCzYwFtQhkBAA/1xrd5+sOSjXIbjfOHPPubNNlCWWkXnocyAgAexjAMPfPlLj2RvUOSdHlaP/3vJSMVFOBncjKgfSgjAOBB3G5D8z7ZoldW7pMk3XrGIM0+e4gsFiYyg+eijACAh3A2uHXnO+v10foCSdKDFw7XtScnm5wK+OUoIwDgAWqcLt30eo6W7zisAD+LnpgyWheP6Wt2LKBDUEYAoJurqmvQ9H+s0eo9ZQoJ9NezV6fp1CG9zI4FdBjKCAB0Y5V1Dbru5e+0Zt8RhVsD9Mp1JzCHCLwOZQQAuil7Tb2uffk75eaVKyI4QK9ef6LGJvY0OxbQ4SgjANANlVc7Ne2l77ThgF22kEC9Pj1DI/vZzI4FdArKCAB0M47a+qYiEhUWpNenZ2h4fKTZsYBOQxkBgG6k2tmg619eow0H7OoZGqg3bzxJQ/tEmB0L6FTtmq5vwYIFSk5OVnBwsNLS0rRixYrj7r9o0SKNHj1aoaGhiouL03XXXafS0tJ2BQYAb1Vb79IN/1irtfuPKCI4QK9Nz6CIwCe0uYwsXrxYs2bN0ty5c5Wbm6uJEydq0qRJysvLa3H/r7/+WtOmTdP06dO1efNmvfPOO1qzZo1uuOGGXxweALyFs8Gt376eo5W7SxUW5K9/XH+iUvsyRgS+oc1l5Mknn9T06dN1ww03KCUlRfPnz1dCQoIWLlzY4v6rV69W//79NXPmTCUnJ+uUU07RTTfdpLVr1/7i8ADgDRpcbt32Vq7+s/2wggP99OK1J2gcd83Ah7SpjDidTuXk5CgzM7PZ9szMTK1cubLFYyZMmKADBw5o6dKlMgxDhw4d0rvvvqvzzz//mD+nrq5ODoej2QMAvJFhGLrvw036dFORgvz99PzV6TppQLTZsYAu1aYyUlJSIpfLpdjY2GbbY2NjVVRU1OIxEyZM0KJFizR16lQFBQWpT58+6tGjh55++ulj/pysrCzZbLamR0JCQltiAoDH+Ev2Dr35Xb78LNJTvx6jXzGzKnxQuwaw/nR1SMMwjrli5JYtWzRz5kzdf//9ysnJ0Weffaa9e/dqxowZx/z+c+bMkd1ub3rk5+e3JyYAdGuvrdqnp77cJUl6eHKqzk2NMzkRYI423dobExMjf3//o86CFBcXH3W25EdZWVk6+eSTddddd0mSRo0apbCwME2cOFGPPPKI4uKO/uOzWq2yWq1tiQYAHmXpxkLd/9FmSdKsswbrqowkkxMB5mnTmZGgoCClpaUpOzu72fbs7GxNmDChxWOqq6vl59f8x/j7+0tqPKMCAL5m1e5SzXprnQxDuiojUbedOdjsSICp2nyZZvbs2XrhhRf00ksvaevWrbr99tuVl5fXdNllzpw5mjZtWtP+F154od5//30tXLhQe/bs0TfffKOZM2fqxBNPVHx8fMf9JgDgAXYVV+qm19bK6XLr3BF9NO/i1GNe5gZ8RZtnYJ06dapKS0s1b948FRYWKjU1VUuXLlVSUuMpxsLCwmZzjlx77bWqqKjQM888ozvuuEM9evTQGWecoT//+c8d91sAgAcorazTda98J0dtg9KSemr+FWPk70cRASyGB1wrcTgcstlsstvtioxkfQYAnqe23qWrXvhWOfuPKDEqVB/8boKiwxkbB+/W2vfvdt1NAwBoPcMwdPe7G5TzwzTvL117AkUE+C+UEQDoZH/51059tL5AAX4WPfebNA3qHW52JKBboYwAQCf6aH2Bnvr3TknSHy8ZqQmDYkxOBHQ/lBEA6CSbDtp197vrJUk3nTpAU05gNmmgJZQRAOgEJZV1uum1HNXWu3Xa0F66+5xhZkcCui3KCAB0sHqXW79b9L0OltcoOSZMf71iLLfwAsdBGQGADjbv4y36bm+Zwq0B+vu0NNlCAs2OBHRrlBEA6EBvfZen11bvl8UizZ86RoN6R5gdCej2KCMA0EE2HCjX/R82Ln53x9lDdNbwlhcQBdAcZQQAOkB5tVO/ff17OV1unT08VjefPsjsSIDHoIwAwC/kdhu6ffE6HSyvUVJ0qB6/fDSL3wFtQBkBgF9owVe79J/th2UN8NOCq8YxYBVoI8oIAPwCX+8s0RPZOyRJD09O1Yh4m8mJAM9DGQGAdiqy12rmW7kyDGlqeoKmpDPDKtAelBEAaAeX29DMt3JVVuXU8LhIPXTxCLMjAR6LMgIA7fDUv3fqu71lCgvy19+uGqfgQH+zIwEeizICAG20anepnv7yh5V4Lx2p5JgwkxMBno0yAgBtUFbl1KzFuXIb0uVp/XTxmL5mRwI8HmUEAFrJMAzd+c56HXLUaWCvMMaJAB2EMgIArfTi13v15bZiBQX46Zkrxyk0KMDsSIBXoIwAQCtsKXDoz59tkyTdd8FwpcRFmpwI8B6UEQD4GbX1Lt32Vq7qXYbOHh6r32Qkmh0J8CqUEQD4GX/6dJt2FleqV4RVf75sFOvOAB2MMgIAx/HV9mK9snKfJOmx/xmlqLAgcwMBXogyAgDHUFpZp7ve3SBJumZ8kk4b2tvkRIB3oowAQAsMw9Cc9zfqcEWdBvUO15zzUsyOBHgtyggAtODttfn6YsshBfpb9NcrxjDdO9CJKCMA8BMHjlTr4U+2SpLuyByqEfE2kxMB3o0yAgD/xe02dPe7G1RZ16C0pJ66ceIAsyMBXo8yAgD/5bXV+7Vyd6lCAv31xOWj5e/HbbxAZ6OMAMAP9pZUKevTxssz90wapv6sxgt0CcoIAEhyuRsXwautd2vCwGhdfVKS2ZEAn0EZAQBJL6zYo5z9RxRuDdCj/zNKflyeAboMZQSAz9tVXKEnsndIku67IEX9eoaanAjwLZQRAD7N9cPdM84Gt04d0ktT0hPMjgT4HMoIAJ/2ysp9+j6vXOHWAGVdOpJF8AATUEYA+Kz9pVV67PNtkqR7z0tRfI8QkxMBvokyAsAnud2Gfv/ehqa7Z359IpdnALNQRgD4pDe+y9PqPWUKCfTXny4dxeUZwESUEQA+52B5jbKWNk5udve5Q5UYzd0zgJkoIwB8imEYmvvBRlU5XUpP6qlrxvc3OxLg8ygjAHzKR+sL9NX2wwry99OfLmNyM6A7oIwA8BlHqpya9/EWSdKtZwzSoN7hJicCIFFGAPiQR/65VaVVTg2NjdBNpw40Ow6AH1BGAPiEr3eW6L3vD8hikbIuG6mgAF7+gO6Cv0YAXq/G6dK9H2yUJE07KUnjEnuanAjAf6OMAPB68/+9Q3ll1YqzBeuuc4eZHQfAT1BGAHi1zQV2vbBiryTpkcmpCrcGmJwIwE9RRgB4LZfb0L0fbJLLbej8kXE6MyXW7EgAWkAZAeC13vh2v9bnlyvCGqD7LxxudhwAx0AZAeCVih21evSz7ZKku84dqtjIYJMTATiWdpWRBQsWKDk5WcHBwUpLS9OKFSuOu39dXZ3mzp2rpKQkWa1WDRw4UC+99FK7AgNAazz0yRZV1DVodD+brspIMjsOgONo80iuxYsXa9asWVqwYIFOPvlkPffcc5o0aZK2bNmixMTEFo+ZMmWKDh06pBdffFGDBg1ScXGxGhoafnF4AGjJV9uL9c8NhfKzSP97yUj5M+U70K1ZDMMw2nJARkaGxo0bp4ULFzZtS0lJ0eTJk5WVlXXU/p999pmuuOIK7dmzR1FRUe0K6XA4ZLPZZLfbFRkZ2a7vAcA31Dhdypy/TPllNZp+SrLuu4CxIoBZWvv+3abLNE6nUzk5OcrMzGy2PTMzUytXrmzxmI8++kjp6el69NFH1bdvXw0ZMkR33nmnampqjvlz6urq5HA4mj0AoDWe/nKn8stqFGcL1uyzh5gdB0ArtOkyTUlJiVwul2Jjm98eFxsbq6KiohaP2bNnj77++msFBwfrgw8+UElJiX73u9+prKzsmONGsrKy9NBDD7UlGgBoV3GF/r5ijyTpwYtGKIw5RQCP0K4BrBZL8+uvhmEcte1HbrdbFotFixYt0oknnqjzzjtPTz75pF555ZVjnh2ZM2eO7HZ70yM/P789MQH4EMMwdN+Szap3GTpzWG9lDmdOEcBTtOl/G2JiYuTv73/UWZDi4uKjzpb8KC4uTn379pXNZmvalpKSIsMwdODAAQ0ePPioY6xWq6xWa1uiAfBxH60v0Ko9pQoO9NODF4045v8gAeh+2nRmJCgoSGlpacrOzm62PTs7WxMmTGjxmJNPPlkFBQWqrKxs2rZjxw75+fmpX79+7YgMAM05auv18CdbJUm3njFYCVGhJicC0BZtvkwze/ZsvfDCC3rppZe0detW3X777crLy9OMGTMkNV5imTZtWtP+V155paKjo3Xddddpy5YtWr58ue666y5df/31CgkJ6bjfBIDPevKLHSqprNOAXmG6YWKy2XEAtFGbR3dNnTpVpaWlmjdvngoLC5WamqqlS5cqKalxUqHCwkLl5eU17R8eHq7s7GzdeuutSk9PV3R0tKZMmaJHHnmk434LAD5r00G7Xl21T5L08MWpsgb4mxsIQJu1eZ4RMzDPCICWuNyGLl3wjdYfsOui0fF66tdjzY4E4L90yjwjANCdLF6Tr/UH7Aq3BugP56eYHQdAO1FGAHiksiqnHv18myRp9tlD1JuF8ACPRRkB4JEe+3ybyqvrNaxPhKaNZyE8wJNRRgB4nHX55XprTeNkiPMuTlWAPy9lgCfjLxiAR3G5Dd3/4SYZhnTpuL46Mbl9C3AC6D4oIwA8yltr8rThgF0R1gDNmcSgVcAbUEYAeIyyKqce/Wy7JOmOzCHqFcGyEYA3oIwA8BiPfrZN9prGQau/OYlBq4C3oIwA8Ajr8su1eG3joNWHJzNoFfAm/DUD6PbcPxm0ekJ/Bq0C3oQyAqDbe3ttvjb8MNPqPZOGmR0HQAejjADo1sqrnfrzZ40zrc46a7B6RzDTKuBtKCMAurUns3foSHW9hsSG65oJ/c2OA6ATUEYAdFubC+x6ffV+SdJDF6UqkEGrgFfiLxtAt2QYhu7/cLPchnTh6HiNHxhtdiQAnYQyAqBb+iD3oHL2H1FokL/uPY9Bq4A3o4wA6HYqauv1x6WNg1ZvOWOQ4mwhJicC0JkoIwC6naf+vVMllXUaEBOm6ackmx0HQCejjADoVnYeqtDL3+yTJN1/4XBZA/zNDQSg01FGAHQbhmHowY83q8Ft6OzhsTptaG+zIwHoApQRAN3Gp5uK9M2uUgUF+On+C4abHQdAF6GMAOgWapwuPfLJFknSjFMHKiEq1OREALoKZQRAt7Dgq10qsNeqb48Q/fbUgWbHAdCFKCMATLe/tErPLdsjSbrvghSFBDFoFfAllBEApnv4ky1yutyaODhG54zoY3YcAF2MMgLAVP/ZVqx/bS1WgJ9FD1w4QhaLxexIALoYZQSAaeoaXHro482SpOtPSdag3uEmJwJgBsoIANO8+PVe7SutVq8Iq249Y5DZcQCYhDICwBSF9ho9/e9dkqR7zxumiOBAkxMBMAtlBIAp/rh0m2rqXUpP6qnJY/qaHQeAiSgjALrc6j2l+nh9gfws0kMXM2gV8HWUEQBdqt7l1gMfNg5avTIjUSPibSYnAmA2ygiALvXaqv3afqhCPUMDdWfmULPjAOgGKCMAuszhijr9JXuHJOnuc4epR2iQyYkAdAeUEQBd5k+fblNFXYNG9bNpSnqC2XEAdBOUEQBdImd/md77/oAkad7FqfL3Y9AqgEaUEQCdzuU2dP8Pg1anpidoTEIPcwMB6FYoIwA63Rvf5WlzgUORwQG6+1wGrQJojjICoFOVVTn1+OfbJUl3ZA5VdLjV5EQAuhvKCIBO9ehn22SvqVdKXKSuykg0Ow6AbogyAqDTfJ93RG+tyZckPXzxCAX485ID4Gi8MgDoFI2DVjdJkv4nrZ/S+0eZnAhAd0UZAdAp3vguT5sOOhQRHKB7Jg0zOw6AbowyAqDDlVbWNQ1avTNzqGIYtArgOCgjADrco59tl72mXsMZtAqgFSgjADrU93lHtHjtD4NWJzNoFcDP41UCQIdpcLl135L/G7SalsSgVQA/jzICoMO8tnq/Nhc4ZAsJ1BwGrQJoJcoIgA5R7KjVE1/skCTdfS4zrQJoPcoIgA7xyD+3qrKuQaMTeujXJzBoFUDrtauMLFiwQMnJyQoODlZaWppWrFjRquO++eYbBQQEaMyYMe35sQC6qW92leij9QXys0j/OzlVfn4WsyMB8CBtLiOLFy/WrFmzNHfuXOXm5mrixImaNGmS8vLyjnuc3W7XtGnTdOaZZ7Y7LIDup67Bpft+mGl12vj+Su1rMzkRAE/T5jLy5JNPavr06brhhhuUkpKi+fPnKyEhQQsXLjzucTfddJOuvPJKjR8/vt1hAXQ/L6zYqz2HqxQTbtXszCFmxwHggdpURpxOp3JycpSZmdlse2ZmplauXHnM415++WXt3r1bDzzwQKt+Tl1dnRwOR7MHgO4nr7RaT3+5U5J03wUpigwONDkRAE/UpjJSUlIil8ul2NjYZttjY2NVVFTU4jE7d+7UPffco0WLFikgIKBVPycrK0s2m63pkZCQ0JaYALqAYRi678NNqq13a8LAaF00Ot7sSAA8VLsGsFoszQenGYZx1DZJcrlcuvLKK/XQQw9pyJDWn76dM2eO7HZ70yM/P789MQF0on9uLNSyHYcV5O+nhyentvgaAACt0bpTFT+IiYmRv7//UWdBiouLjzpbIkkVFRVau3atcnNzdcstt0iS3G63DMNQQECAvvjiC51xxhlHHWe1WmW1MkcB0F05auv10MdbJEm/PW2gBvYKNzkRAE/WpjMjQUFBSktLU3Z2drPt2dnZmjBhwlH7R0ZGauPGjVq3bl3TY8aMGRo6dKjWrVunjIyMX5YegCke/3y7DlfUaUBMmH572kCz4wDwcG06MyJJs2fP1tVXX6309HSNHz9ezz//vPLy8jRjxgxJjZdYDh48qFdffVV+fn5KTU1tdnzv3r0VHBx81HYAnmFdfrleW71fkvTIJakKDvQ3OREAT9fmMjJ16lSVlpZq3rx5KiwsVGpqqpYuXaqkpCRJUmFh4c/OOQLAMzW43Lr3/Y0yDOnSsX01YWCM2ZEAeAGLYRiG2SF+jsPhkM1mk91uV2RkpNlxAJ/1woo9euSfW2ULCdS/7zhVMaw/A+A4Wvv+zdo0AFolv6y6aSG8OZOGUUQAdBjKCICfZRiG5i7ZpJp6lzKSozQlnbl/AHQcygiAn7Vk3UEt33FYQQF+yrp0JAvhAehQlBEAx1VW5dTDn2yVJM08Y5AGMKcIgA5GGQFwXI98skVlVU4NjY3Q//sVc4oA6HiUEQDHtHzHYb2fe1AWi/Sny0YqKICXDAAdj1cWAC2qdjZo7pKNkqRrxvfX2MSeJicC4K0oIwBa9PjnO5RfVqN4W7DuPGeo2XEAeDHKCICj5Owv08sr90qS/vfSkQq3tnmyZgBoNcoIgGZq6126690NMgzpsnH9dPrQ3mZHAuDlKCMAmnnq3zu153CVekVYdd8FKWbHAeADKCMAmmw8YNdzy/dIkh6ZnKoeoUEmJwLgCygjACRJzga37np3vVxuQxeMitM5I/qYHQmAj6CMAJAkLfhql7YVVSgqLEgPXTTC7DgAfAhlBIA2HbTrmS93SZIeuHC4olmRF0AXoowAPq6uwaU73l6vBrehSal9dNHoeLMjAfAxlBHAx/31Xzu1/VCFosOC9MjkVFksrMgLoGtRRgAf9n3eET27bLck6Y+XjuTyDABTUEYAH1XjdOnOt9fLbUiXju3L3TMATEMZAXzUo59v056SKsVGWvXAhdw9A8A8lBHAB63cXaKXv9knSfrzZaNkCw00NxAAn0YZAXyMvbped7y9XpL06xMTdRprzwAwGWUE8CGGYWjuko0qtNcqOSaMtWcAdAuUEcCHLFl3UJ9sKJS/n0V/mTpGoUEBZkcCAMoI4Cvyy6p1/5LNkqRZZw7WmIQe5gYCgB9QRgAf4HIbuuPt9aqoa1BaUk/99rSBZkcCgCaUEcAHPLtst77bV6Zwa4DmTx2jAH/+9AF0H7wiAV4uZ/8RPZm9Q5L04EUjlBAVanIiAGiOMgJ4MXtNvWa+mSuX29CFo+N12bi+ZkcCgKNQRgAvZRiG7nlvgw6W1ygxKlR/vIRF8AB0T5QRwEst+jZPn24qUqC/Rc9cOVYRwcyyCqB7oowAXmhroUPzPtkiSfr9ucM0ql8PcwMBwHFQRgAvU+1s0C1vfC9ng1unD+2l609ONjsSABwXZQTwIoZh6A9LNmn34cbVeB+/fLT8/BgnAqB7o4wAXuTN7/L1/vcH5e9n0V+vGKvocKvZkQDgZ1FGAC+x8YBdD37UON37XecM1UkDok1OBACtQxkBvEB5tVO/XZQjp8uts4fH6qZfDTA7EgC0GmUE8HBut6HZb6/XgSM1SooO1eOXj2Y+EQAehTICeLiFy3bry23Fsgb4acFV42QLYT4RAJ6FMgJ4sK+2F+vxL7ZLkh6+OFUj4m0mJwKAtqOMAB5qz+FK3fpmrgxD+vWJiZpyQoLZkQCgXSgjgAeqqK3Xja+uVUVtg9KTeuqhi0aYHQkA2o0yAngYt9vQ7YvXaffhKvWJDNaC34xTUAB/ygA8F69ggIf5y7926F9bGwesPj8tTb0jgs2OBAC/CGUE8CD/3FCop7/cJUn602UjWQAPgFegjAAeIjfviGa/vU6SdMMpybpkbD9zAwFAB6GMAB4gv6xaN766VnUNbp0xrLfmnJdidiQA6DCUEaCbc9TWa/o/1qik0qmUuEg99eux8mclXgBehDICdGP1LrduXvS9dhyqVGykVS9dm65wa4DZsQCgQ1FGgG7KMAw9+NFmrdhZopBAf714zQmKs4WYHQsAOhxlBOimFny1W4u+zZPFIv31ijFK7ctU7wC8U7vKyIIFC5ScnKzg4GClpaVpxYoVx9z3/fff19lnn61evXopMjJS48eP1+eff97uwIAveOu7PD32eeOaMw9cMFyZI/qYnAgAOk+by8jixYs1a9YszZ07V7m5uZo4caImTZqkvLy8Fvdfvny5zj77bC1dulQ5OTk6/fTTdeGFFyo3N/cXhwe8UfaWQ7r3g42SpN+dNlDXnpxsciIA6FwWwzCMthyQkZGhcePGaeHChU3bUlJSNHnyZGVlZbXqe4wYMUJTp07V/fff36r9HQ6HbDab7Ha7IiMj2xIX8Chr95Xpqhe+VV2DW1PS++nPl42SxcKdMwA8U2vfv9t0ZsTpdConJ0eZmZnNtmdmZmrlypWt+h5ut1sVFRWKioo65j51dXVyOBzNHoC323GoQte/skZ1DW6dOay3/njJSIoIAJ/QpjJSUlIil8ul2NjYZttjY2NVVFTUqu/xxBNPqKqqSlOmTDnmPllZWbLZbE2PhASWRod321tSpate+FaO2gaNS+yhZ64cpwB/xpcD8A3terX76f+tGYbRqv+De/PNN/Xggw9q8eLF6t279zH3mzNnjux2e9MjPz+/PTEBj3DgSLWu+vtqHa6o07A+EXrp2hMUEuRvdiwA6DJtmj0pJiZG/v7+R50FKS4uPupsyU8tXrxY06dP1zvvvKOzzjrruPtarVZZrda2RAM80iFHra78+7cqsNdqQK8wvTY9Qz1Cg8yOBQBdqk1nRoKCgpSWlqbs7Oxm27OzszVhwoRjHvfmm2/q2muv1RtvvKHzzz+/fUkBL1NSWacr/75aeWXVSowK1Rs3nKReEZRwAL6nzfNKz549W1dffbXS09M1fvx4Pf/888rLy9OMGTMkNV5iOXjwoF599VVJjUVk2rRp+utf/6qTTjqp6axKSEiIbDYmcYJvOlLl1NUvfqfdh6sUbwvWohsy1McWbHYsADBFm8vI1KlTVVpaqnnz5qmwsFCpqalaunSpkpKSJEmFhYXN5hx57rnn1NDQoJtvvlk333xz0/ZrrrlGr7zyyi//DQAPU1JZp9+88K22FVWoV4RVi248SQlRoWbHAgDTtHmeETMwzwi8RbGjVle+8K12FVeqd4RVb9x4kgb1Djc7FgB0ita+f7P8J9BFiuy1uvLvq7WnpEpxtmC9ceNJSo4JMzsWAJiOMgJ0gYPlNbry76u1v7RafXuE6M0bT1JiNJdmAECijACdbldxpaa92Hj7bmJUqN64MUP9elJEAOBHlBGgE63LL9d1L3+nI9X1GtArTItuyFCcLcTsWADQrVBGgE6yfMdhzXg9R9VOl0b3s+nl605UVBgTmgHAT1FGgE7w0foC3fH2OtW7DE0cHKNnf5OmMCt/bgDQEl4dgQ5kGIZeWLFXf/x0qwxDumBUnJ6YMlrWANaaAYBjoYwAHaTe5dYDH23WG982Tvo3bXySHrhwhPz9fn4RSQDwZZQRoAM4aut186LvtWJniSwW6Q/nD9f1J/dv1WrWAODrKCPAL5RfVq3rX1mjncWVCgn011O/Hquzhx9/FWsAwP+hjAC/wMpdJbrlzVyVVTkVG2nVi9ecoNS+LAAJAG1BGQHa4ceBqlmfbpXbkFL7Rurv09KZQwQA2oEyArRRtbNBv39voz5eXyBJunRcX/3xkpEKDuSOGQBoD8oI0AZ7S6r029dztK2oQgF+Ft13wXBNG5/EQFUA+AUoI0ArLck9qLkfbFSV06WYcKsWXDVOJyZHmR0LADweZQT4GdXOBj3w4Wa9k3NAkpSRHKWnfj1WsZHBJicDAO9AGQGOY1uRQ7e8katdxZXys0i3njFYM88czERmANCBKCNAC1xuQy+s2KMnvtghp8ut2Eir5k8dq/EDo82OBgBehzIC/MT+0ird+c56rdl3RJJ0xrDeeux/Rik63GpyMgDwTpQR4AeGYWjRt3n649Ktqna6FBbkr/suGK6pJyRwtwwAdCLKCKDGW3bnfrBRK3eXSmocpPr45aOVEBVqcjIA8H6UEfg0Z4Nbzy/frae+3CVng1vBgX6665xhum5Cf/kxSBUAugRlBD4rZ/8RzXl/g3YcqpQkTRwco/+dPFKJ0ZwNAYCuRBmBzyl21OrPn23Xe983zhsSFRak+y8YrovHxDM2BABMQBmBz3A2uPXyN3v11L93qsrpkiRdntZP956Xop5hQSanAwDfRRmB1zMMQ9lbDinr023aW1IlSRqd0EMPXjhcYxN7mpwOAEAZgVdbs69Mf/p0m3L2N84ZEhNu1T2ThunSsX0ZoAoA3QRlBF5px6EKPfrZNv1ra7EkKTjQT9efnKzfnjZQEcGBJqcDAPw3ygi8yo5DFXrq3zv1z42FMgzJ38+iqSck6LYzB7OwHQB0U5QReIWthQ49/eVOLd1Y1LRtUmof3XnOUA3sFW5iMgDAz6GMwGMZhqG1+4/ouWV79K+th5q2T0rto5lnDlZKXKSJ6QAArUUZgcdxuQ19vrlIzy/fo3X55ZIki0U6b2ScZp4xWEP7RJgbEADQJpQReIzyaqfeWXtAr63er7yyaklSUICfLhvXV9NPGaBBvbkcAwCeiDKCbm/jAbteXbVPH60vUF2DW5LUIzRQ005K0tXj+6tXhNXkhACAX4Iygm7JXlOvj9cX6J21+Vp/wN60PSUuUtPGJ+niMfEKDeKfLwB4A17N0W243IZW7ynVO2vz9emmoqazIIH+Fk1KjdO08UlKS+rJ+jEA4GUoIzCVYRjaeNCuD9cV6JMNBTrkqGv62pDYcE1JT9DksX0VE86lGADwVpQRdDnDMLS5wKHPNxfpkw2FTevFSFJkcIAuHB2vKekJGtXPxlkQAPABlBF0CZfbUG7eEX22qUifbS7SgSM1TV8LDvTTWSmxumh0vE4d2kvWAH8TkwIAuhplBJ2mvNqp5TtL9J9txVq247DKqpxNXwsO9NOpQ3ppUmqczh4eqzAr/xQBwFfxDoAOU+9ya31+ub7eVaKvd5bo+7wjchv/9/WI4ACdlRKrc0b00alDeikkiDMgAADKCH6BBpdbWwod+m5vmVbtLtXqPaWqcrqa7TM0NkKnDeulM4b2VlpSTwX4+5mUFgDQXVFG0GpVdQ1af6BcuXnl+nZvmXL2lR1VPnqGBmrCoBhNHBSjUwbHqF/PUJPSAgA8BWUELWpwubWzuFIbD9q1Pr9c3+eVa3uRo9llF6nx0suJ/aN0YnKUTh4Uo+FxkfLz4w4YAEDrUUag2nqXthdVaGuhQ1sKHdp40K6thQ7V1ruP2jfeFqyxST2VntRTGcnRGtonQv6UDwDAL0AZ8SH1Lrf2l1Zpx6FK7ThUoZ2HKrWtyKG9JVVHnfGQpHBrgFL7RmpkX5vGJfbU2MSe6mML7vrgAACvRhnxMoZhqLiiTvtKqrSvtEp7Dldp9+Eq7SmpVF5ptRpaah2SosOClBIXqZS4CKX2tSm1r03J0WFccgEAdDrKiIcxDEOOmgYdLK/RgSPVOnCkRgeO1Cj/SLXySqu1v6yqxcsrPwoL8teg2AgN6R2uwbHhGhIboeFxkeoVYWW2UwCAKSgj3YjLbaisyqlDjloVV9SqyF6nQ45aFdlrVWCvUaG9VoXlNUfdwfJT/n4WxfcIVv/oMA3sFa4BvcI0IKbxY5wtmNIBAOhWKCOdyDAMVdQ1qLyqXmXVTh2pcqq0yqnSyjqVVjlVUlmn0kqnDlfU6XBlnUor61ocu9GSqLAgJfQMUb+eoerXM0T9eoYoMTpMSVGh6tszRIHM5wEA8BDtKiMLFizQY489psLCQo0YMULz58/XxIkTj7n/smXLNHv2bG3evFnx8fG6++67NWPGjHaH7iqGYaiuwa3KugZV1jaoorZBFbX1cvzXR0dNvRy19XLUNMhe41R5db3Ka+pVXl0ve41T9a5WtosfWCxSdJhVfWxWxUYEK9YWrNiIYMX1CFa8LUTxPYIVZwth9lIAgNdocxlZvHixZs2apQULFujkk0/Wc889p0mTJmnLli1KTEw8av+9e/fqvPPO04033qjXX39d33zzjX73u9+pV69euuyyyzrkl2iv11fv1/f7j6jK2aBqp0uVdQ2qrmv8WOVsLCDHGvDZFiGB/ooKC1KP0EBFh1sVExak6PAgRYVZFR0epF4RVvUKt6p3hFVRYUHMUgoA8CkWwzDa9G6bkZGhcePGaeHChU3bUlJSNHnyZGVlZR21/+9//3t99NFH2rp1a9O2GTNmaP369Vq1alWrfqbD4ZDNZpPdbldkZGRb4h7XzDdz9dH6glbtGxbkr4jgQIUHBygiOEARwYGyhQQqMjhAkSGBigwOVI/QQPUICZQtNFA9QhrLR8/QIM5iAAB8Umvfv9t0ZsTpdConJ0f33HNPs+2ZmZlauXJli8esWrVKmZmZzbadc845evHFF1VfX6/AwMCjjqmrq1NdXV2zX6YzXDg6Xql9IxUaFKAwq3/jxx/+O9waoPDgAIVZG7cxsRcAAJ2jTWWkpKRELpdLsbGxzbbHxsaqqKioxWOKiopa3L+hoUElJSWKi4s76pisrCw99NBDbYnWLmcPj5UU+7P7AQCAztOuwQk/vTXUMIzj3i7a0v4tbf/RnDlzZLfbmx75+fntiQkAADxAm86MxMTEyN/f/6izIMXFxUed/fhRnz59Wtw/ICBA0dHRLR5jtVpltVrbEg0AAHioNp0ZCQoKUlpamrKzs5ttz87O1oQJE1o8Zvz48Uft/8UXXyg9Pb3F8SIAAMC3tPkyzezZs/XCCy/opZde0tatW3X77bcrLy+vad6QOXPmaNq0aU37z5gxQ/v379fs2bO1detWvfTSS3rxxRd15513dtxvAQAAPFab5xmZOnWqSktLNW/ePBUWFio1NVVLly5VUlKSJKmwsFB5eXlN+ycnJ2vp0qW6/fbb9be//U3x8fF66qmnTJ9jBAAAdA9tnmfEDJ01zwgAAOg8rX3/ZqpPAABgKsoIAAAwFWUEAACYijICAABMRRkBAACmoowAAABTUUYAAICpKCMAAMBUlBEAAGAqyggAADAVZQQAAJiKMgIAAExFGQEAAKaijAAAAFNRRgAAgKkoIwAAwFSUEQAAYCrKCAAAMBVlBAAAmIoyAgAATEUZAQAApqKMAAAAU1FGAACAqQLMDtAahmFIkhwOh8lJAABAa/34vv3j+/ixeEQZqaiokCQlJCSYnAQAALRVRUWFbDbbMb9uMX6urnQDbrdbBQUFioiIkMViMTuO6RwOhxISEpSfn6/IyEiz43g1nuuuw3PddXiuu46vP9eGYaiiokLx8fHy8zv2yBCPODPi5+enfv36mR2j24mMjPTJf9xm4LnuOjzXXYfnuuv48nN9vDMiP2IAKwAAMBVlBAAAmIoy4oGsVqseeOABWa1Ws6N4PZ7rrsNz3XV4rrsOz3XreMQAVgAA4L04MwIAAExFGQEAAKaijAAAAFNRRgAAgKkoI16irq5OY8aMkcVi0bp168yO43X27dun6dOnKzk5WSEhIRo4cKAeeOABOZ1Os6N5jQULFig5OVnBwcFKS0vTihUrzI7kdbKysnTCCScoIiJCvXv31uTJk7V9+3azY/mErKwsWSwWzZo1y+wo3RJlxEvcfffdio+PNzuG19q2bZvcbreee+45bd68WX/5y1/07LPP6t577zU7mldYvHixZs2apblz5yo3N1cTJ07UpEmTlJeXZ3Y0r7Js2TLdfPPNWr16tbKzs9XQ0KDMzExVVVWZHc2rrVmzRs8//7xGjRpldpRui1t7vcCnn36q2bNn67333tOIESOUm5urMWPGmB3L6z322GNauHCh9uzZY3YUj5eRkaFx48Zp4cKFTdtSUlI0efJkZWVlmZjMux0+fFi9e/fWsmXL9Ktf/crsOF6psrJS48aN04IFC/TII49ozJgxmj9/vtmxuh3OjHi4Q4cO6cYbb9Rrr72m0NBQs+P4FLvdrqioKLNjeDyn06mcnBxlZmY2256ZmamVK1ealMo32O12SeLfcSe6+eabdf755+uss84yO0q35hEL5aFlhmHo2muv1YwZM5Senq59+/aZHcln7N69W08//bSeeOIJs6N4vJKSErlcLsXGxjbbHhsbq6KiIpNSeT/DMDR79mydcsopSk1NNTuOV3rrrbeUk5OjtWvXmh2l2+PMSDf04IMPymKxHPexdu1aPf3003I4HJozZ47ZkT1Wa5/r/1ZQUKBzzz1Xl19+uW644QaTknsfi8XS7HPDMI7aho5zyy23aMOGDXrzzTfNjuKV8vPzddttt2nRokUKDg42O063x5iRbqikpEQlJSXH3ad///664oor9PHHHzd7wXa5XPL399dVV12lf/zjH50d1eO19rn+8cWkoKBAp59+ujIyMvTKK6/Iz48+/0s5nU6FhobqnXfe0SWXXNK0/bbbbtO6deu0bNkyE9N5p1tvvVVLlizR8uXLlZycbHYcr7RkyRJdcskl8vf3b9rmcrlksVjk5+enurq6Zl/zdZQRD5aXlyeHw9H0eUFBgc455xy9++67ysjIUL9+/UxM530OHjyo008/XWlpaXr99dd5IelAGRkZSktL04IFC5q2DR8+XBdffDEDWDuQYRi69dZb9cEHH+irr77S4MGDzY7ktSoqKrR///5m26677joNGzZMv//977k09hOMGfFgiYmJzT4PDw+XJA0cOJAi0sEKCgp02mmnKTExUY8//rgOHz7c9LU+ffqYmMw7zJ49W1dffbXS09M1fvx4Pf/888rLy9OMGTPMjuZVbr75Zr3xxhv68MMPFRER0TQmx2azKSQkxOR03iUiIuKowhEWFqbo6GiKSAsoI0ArfPHFF9q1a5d27dp1VNHj5OIvN3XqVJWWlmrevHkqLCxUamqqli5dqqSkJLOjeZUfb50+7bTTmm1/+eWXde2113Z9IOAHXKYBAACmYvQdAAAwFWUEAACYijICAABMRRkBAACmoowAAABTUUYAAICpKCMAAMBUlBEAAGAqyggAADAVZQQAAJiKMgIAAExFGQEAAKb6/1huQd2sFXHqAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X = np.arange(-5.0, 5.0, 0.1)\n",
    "Y = sigmoid(X)\n",
    "plt.plot(X, Y)\n",
    "plt.ylim(-0.1, 1.1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6249253a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    exp_x = np.exp(x)\n",
    "    sum_exp_x = np.sum(exp_x)\n",
    "    y = exp_x/sum_exp_x\n",
    "\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "b19b49e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA+f0lEQVR4nO3deViVdf7/8ddhO+yooAgKiPuCuUCZltNO2Wr1TZuabLF+OVOZ2TJZ06I1w0xTTVONVtPiVFa2OtXYwkyTWWopornvCgqIoHLYD5xz//5AGUk0QOC+OffzcV3nQm7uc84bLj+cF5/P+3xuh2EYhgAAAEziZ3YBAADA3ggjAADAVIQRAABgKsIIAAAwFWEEAACYijACAABMRRgBAACmIowAAABTBZhdQFN4vV7l5eUpIiJCDofD7HIAAEATGIah0tJSxcfHy8/v2PMfHSKM5OXlKSEhwewyAABAC+Tm5qpnz57H/HqHCCMRERGS6r6ZyMhIk6sBAABN4XK5lJCQUP86fiwdIowcXpqJjIwkjAAA0MH8XIsFDawAAMBUhBEAAGAqwggAADAVYQQAAJiKMAIAAExFGAEAAKYijAAAAFMRRgAAgKkIIwAAwFSEEQAAYCrCCAAAMBVhBAAAmIowAgAATEUYAQAApiKMAAAAUxFGAACAqQgjAADAVIQRAABgKsIIAAAwFWEEAACYijACAABMRRgBAACmIowAAABTNTuMfPPNN7rkkksUHx8vh8OhBQsW/Ox9Fi1apNTUVAUHB6t379564YUXWlIrAADwQc0OI+Xl5Ro2bJief/75Jp2/Y8cOXXjhhRo7dqyys7P1wAMPaOrUqfrggw+aXSwAAPA9Ac29w7hx4zRu3Lgmn//CCy8oMTFRzzzzjCRp0KBBWrFihZ588kldeeWVzX16AADgY5odRppr6dKlSk9Pb3Ds/PPP1yuvvKKamhoFBgYedZ/q6mpVV1fXf+5yudq6TABWU1Ul/eUv0t69ZlcC2MOkSdLIkaY8dZuHkYKCAsXGxjY4Fhsbq9raWhUVFSkuLu6o+2RkZGjmzJltXRoAK/vXv6QHHjC7CsA+Tj3Vd8OIJDkcjgafG4bR6PHDZsyYoenTp9d/7nK5lJCQ0HYFArCekpK6j336SBMnmlsL4CMMw1Ct15Db41VNrVdujyF3rVc1Hq8iE/so2qS62jyMdO/eXQUFBQ2OFRYWKiAgQNHRjX/bTqdTTqezrUsDYGUeT93HIUOk3//e3FoACzEMQ66qWh0od+tgZY0OVrhVUlmjgxV1t5LKGrmqauSq/1ir0uoalVbVqqyqVrUyJH/V3Y7wt/CeusiU76gdwsjo0aP1ySefNDj25ZdfKi0trdF+EQCQJNXW1n0MaJcJXMBUVTUeFZVVa1/poVtZtYpK3Sour1ZxuVvFZdUqLnPrQEVd+Kj1Gif0fA6HFB4UoDBngMKc/gp3BijU6f/zd2wjzR7lZWVl2rp1a/3nO3bs0KpVq9SlSxclJiZqxowZ2rNnj15//XVJ0pQpU/T8889r+vTpuuWWW7R06VK98sorevvtt1vvuwDgewgj8BGlVTXKL6lS3sFK5ZdUKf/Qx72l1dpbUqW9pVU6WFHT7McNDfJX59AgdQoNrLuFBCkyJFBRh26RIQGKDA5URHCAIoIDFXnoY3hwgEID/eXn13irhBmaPcpXrFihs846q/7zw70d119/vebOnav8/Hzl5OTUfz05OVkLFy7UXXfdpb/97W+Kj4/Xs88+y9t6ARzf4TDib95fa0BTVNd6lLu/Ujn7y7WruEK5+yu1+0CFdh+o++iqqm3S4wT5+6lrhLP+FhMepJhwp7qEBSk63KnosCB1CQuqDyDBgb4zNpodRs4888z6BtTGzJ0796hjZ5xxhlauXNncpwJgZ4d7RpgZgQV4vYb2HKzU9qJybd9Xpu37yrW9qEw7iyqUV1Kp47wsSpKiQgIVFxWs+E4hiosKVlxUsGIjj7w5FRUSeMw3dvg6RjkAa2KZBiYwDEOFpdVan+/S5oJSbd5bpi2FpdpaWKYKt+eY9wsL8ldidJiSuoQqMTpUPTuHHLqFqkenEIU5+X98PPx0AFgTYQRtzOs1tLO4XGv2lGjtnhKtz3dpQ36p9pe7Gz0/0N+hXtFh6t01TMkx4erdNUy9Y8KUFB2mmPAg285qtAZGOQBrIoyglRW6qrQy56Cycw5o9e6DWrfHpdLqo/s5/BxS767hGtg9Qv1jI9SvW7j6xUaoV3SoAvy52H1bYJQDsCYaWHECvF5DWwrL9P2OYi3feUArdx3QnoOVR53nDPDT4PhIDe0RpSHxkRoUF6n+sRE+1RzaERBGAFgTDaxoBsMwtHlvmb7dWqRl24u1fOf+o94u63BIA2IjNCKxs4YnROmknp3Ur1s4sx0WwCgHYE0s0+Bn7HVV6ZvN+/Td1iJ9u7VYRWXVDb4eEuiv1KTOOrlXF6X16qyTekYpIpjNNq2IUQ7Amggj+AmP19Dq3Qf1342F+mpjodblNbyie3Cgn0YlR2t0n2iNSu6ilB5RCmTWo0NglAOwJsIIVLeh2JJtxfpibYEy1+9V8RHvdHE4pJN6RGlsv646rW+MRiZ1kjOAXo+OiFEOwJpoYLWtqhqPvt60T5+tzddXGwobvOMlIjhAv+jfVWcN6KYzB3RVTDgXVfUFhBEA1kQDq63Uerxaur1YH6/K0+drCxoEkNhIp9IHd9cFKd11SnIXll58EKMcgDWxTGMLW/aW6r2s3fooe4/2lf6vATUuKlgXnxSncUPjNLxnJ0td1A2tj1EOwJoIIz6rtKpGn/6Yr3dX5Co752D98c6hgbpwaJwuG95DaUmdCSA2wigHYE30jPicjQUuvbF0lz7K3lN/nRd/P4fOHthNV6X21FkDu7EEY1OEEQDWRM+IT6jxePXZ2gK9uXSXfti5v/54327hmpiWoPEjeqhrBE2odscoB2BNLNN0aKVVNZq/PFevfrtDeSVVkupmQS4Y0l2/OjVJp/buwoXlUI9RDsCaCCMdUkFJlV5bskNvLcupf0dMTHiQrh2VpGtGJSo2MtjkCmFFjHIA1kQY6VD2HKzUnK+36t3lu+X2eCVJfbqG6ZaxvTV+RA8uPIfjYpQDsCYaWDuE3P0Vmv31Nr2flasajyFJOrlXZ936iz46e2A33hGDJiGMALAmGlgtrdBVpb/+Z4vmL89VrbcuhIzuHa07z+2nU3tHm1wdOhpGOQBrYpnGkkqravTSN9v18uIdqqypC4yn943R1HP66ZTkLiZXh46KUQ7AmggjllLj8eqNpbv03FdbdKCiRpI0IrGT7r9goEYxE4ITxCgHYE2EEctYtHmfZn2yTtv2lUuSencN033nD9T5Q2J5ey5aBaMcgDXRwGq6nUXlevxf6/XvDYWSpOiwIN2dPkAT0noqgJ1S0YoIIwCsiQZW01TVePS3/27Vi4u2y+3xKsDPoevH9NLUc/opKiTQ7PLggxjlAKyJZRpTLNlWpAc/WqsdRXVLMr/o31UPXzxIfbtFmFwZfBmjHIA1EUba1YFyt/6wcIPey9otSeoW4dTMS4fogpTu9IWgzTHKAVgTPSPt5vO1BfrdgjUqKnPL4ZB+NSpJ914wQJHBLMmgfRBGAFgTPSNtrqSyRjM/XqcPs/dIkvrHhivjiqFKTWK/ELQvRjkAa2KZpk0t3rJP973/o/JLquTnkKac0Ud3nttPzgBmotD+GOUArIkw0iaqajzKWLhB/1i6S5LUKzpUT00YrtSkziZXBjtjlAOwJsJIq9u+r0y3v5Wt9fkuSdKk0Um6f9xAhQbxM4a5+B8IwJpoYG1VH67crd8tWKsKt0ddwoL01IRhOmtAN7PLAiQRRgBYFQ2sraLCXavfLVirD1fWNamO7h2tZ64ertjIYJMrA/6HUQ7AmlimOWG7ist16xtZ2lhQKj+HNO3c/rrtrL7y92PfEFgLoxyANRFGTsjXmwo19e1suapqFRPu1PPXjNCpXF0XFsUoB2BNhJEWMQxDs7/epie/3CTDkIYndNILv0pV9yiWZWBdjHIA1mMYktdb928aWJuswl2ru99drc/WFkiSfnlKoh69dDB7h8DyCCMArOdw86rEzEgTFbqqNPkfK7RmT4mC/P0087Ih+uUpiWaXBTQJoxyA9RxeopEII02wId+lyXOXK6+kSl3CgvTSdalK68WW7ug4GOUArIcw0mT/3Vio299aqXK3R727hum1G05WUnSY2WUBzcIoB2A9R4YRekaO6a3vc/S7BWvkNer2D3nhV6mKCuVKu+h4CCMArIeekeMyDEPPf7VVT2VuliRdldpTv798qIIC/EyuDGgZRjkA6zlyZsSPF9gjeb2GZn26XnOX7JQk3XF2X00/r78cDjYyQ8dFGAFgPUfuMcKLbD13rVf3vLdaH6/OkyQ9eslg3XBasslVASeOMALAetjw7CiVbo9ufTNL32zepwA/h56aMEyXDe9hdllAq2CkA7AertjbQHl1rSb/Y7mWbd+vkEB/vXBdqs7o39XssoBWQxgBYD1csbdeWXWtbnztBy3feUDhzgDNvfFk9hCBz2GkA7AelmkkSSWVNbrhtR+UnXNQEcEBev2mUzQisbPZZQGtzt4jHYA1EUZ0sMKtSa/+oB93lygqJFBvTh6loT2jzC4LaBP2HekArMvmYcRVVVMfRLqEBenNyaM0OD7S7LKANmPPkQ7A2mzcwFrhrtVNry3Xj7tL1Dk0UG/fcqoGdI8wuyygTbVoN6HZs2crOTlZwcHBSk1N1eLFi497/rx58zRs2DCFhoYqLi5ON954o4qLi1tUMAAbsGkDa1WNRzf/Y4VW7DqgiOAAvTF5FEEEttDsMDJ//nxNmzZNDz74oLKzszV27FiNGzdOOTk5jZ7/7bffatKkSZo8ebLWrVun9957T8uXL9fNN998wsUD8FE2XKZx13r16zeztGRbscKC/PWPm05RSg96RGAPzQ4jTz/9tCZPnqybb75ZgwYN0jPPPKOEhATNmTOn0fOXLVumXr16aerUqUpOTtbpp5+uW2+9VStWrDjh4gH4KJuFkVqPV3e+k63/btqn4EA/vXLDyRrJu2ZgI80KI263W1lZWUpPT29wPD09XUuWLGn0PmPGjNHu3bu1cOFCGYahvXv36v3339dFF110zOeprq6Wy+VqcANgIzbqGTEMQw/9c60+W1ugIH8/vXRdmk7tHW12WUC7alYYKSoqksfjUWxsbIPjsbGxKigoaPQ+Y8aM0bx58zRx4kQFBQWpe/fu6tSpk5577rljPk9GRoaioqLqbwkJCc0pE0BHZ6Oekb9kbtbbP+TKzyE9+8vh+gU7q8KGWtTA+tOrQxqGccwrRq5fv15Tp07Vww8/rKysLH3++efasWOHpkyZcszHnzFjhkpKSupvubm5LSkTQEdlk2WaN5bu1LNfbZUkPTY+RRekxJlcEWCOZo30mJgY+fv7HzULUlhYeNRsyWEZGRk67bTTdO+990qSTjrpJIWFhWns2LF6/PHHFRd39OBzOp1yOp3NKQ2AL7FBGFm4Jl8Pf7xOkjTt3H66dlSSyRUB5mnWzEhQUJBSU1OVmZnZ4HhmZqbGjBnT6H0qKirk59fwafwPrQMbhtGcpwdgFz4eRpZuK9a0d1bJMKRrRyXqznP6mV0SYKpmL9NMnz5dL7/8sl599VVt2LBBd911l3JycuqXXWbMmKFJkybVn3/JJZfoww8/1Jw5c7R9+3Z99913mjp1qk455RTFx8e33ncCwHf4cAPr1sIy3frGCrk9Xl0wpLtmXZZyzGVuwC6a/WfHxIkTVVxcrFmzZik/P18pKSlauHChkpLqphjz8/Mb7Dlyww03qLS0VM8//7zuvvtuderUSWeffbb+9Kc/td53AcC3+GgDa3FZtW6c+4NcVbVKTeqsZ64eLn8/ggjgMDrAWonL5VJUVJRKSkoUGcn1GQCf949/SDfcIF1wgfTZZ2ZX0yqqajy69uXvlbXrgBK7hOqj34xRdDi9cfBtTX39btG7aQCgTflYz4hhGLrv/R+VdWib91dvOJkgAhyBMALAenwsjPzl31v08eo8Bfg59OKvUtW3W7jZJQGWQhgBYD0+1MD68eo8PfufLZKkP1w+VGP6xphcEWA9hBEA1uMjDaxr95TovvdXS5JuPaO3JpzMbtJAYwgjAKzHB5ZpisqqdesbWaqq8erMAV113/kDzS4JsCzCCADr6eBhpMbj1W/mrdSeg5VKjgnTX68ewVt4geMgjACwng7eMzLrk/X6Ycd+hTsD9PdJqYoKCTS7JMDSCCMArKcD94y880OO3li2Sw6H9MzE4erbLcLskgDLI4wAsJ4Oukzz4+6DevifdRe/u/u8/jp3cOMXEAXQEGEEgPV0wDBysMKtX7+5Um6PV+cNjtVtZ/U1uySgwyCMALCeDhZGvF5Dd81fpT0HK5UUHaonrxrGxe+AZiCMALCeDtbAOvvrrfrvpn1yBvhp9rUjaVgFmokwAsB6OlAD67dbivRU5mZJ0mPjUzQkPsrkioCOhzACwHo6yDJNQUmVpr6TLcOQJqYlaEIaO6wCLUEYAWA9HSCMeLyGpr6Trf3lbg2Oi9TMy4aYXRLQYRFGAFhPBwgjz/5ni37YsV9hQf7627UjFRzYMfpbACsijACwHos3sC7dVqznvjp0Jd4rhio5JszkioCOjTACwHos3MC6v9ytafOz5TWkq1J76rLhPcwuCejwCCMArMeiyzSGYeie91Zrr6tafbqG0ScCtBLCCADrsWgYeeXbHfpqY6GCAvz0/DUjFRpkrfqAjoowAsB6LNgzsj7PpT99vlGS9NDFgzUoLtLkigDfQRgBYD0W6xmpqvHozneyVeMxdN7gWP1qVKLZJQE+hTACwHostkzzx882akthmbpGOPWnK0/iujNAKyOMALAeC4WRrzcVau6SnZKkP//fSeoSFmRuQYAPIowAsB6LhJHismrd+/6PkqTrRyfpzAHdTK0H8FWEEQDWY4EGVsMwNOPDNdpXWq2+3cI148JBptUC+DrCCADrsUAD67srcvXl+r0K9Hfor1cPZ7t3oA0RRgBYj8nLNLsPVOixTzdIku5OH6Ah8VGm1AHYBWEEgPWYGEa8XkP3vf+jyqprlZrUWbeM7d3uNQB2QxgBYD0m9oy8sWyXlmwrVkigv566apj8/XgbL9DWCCMArMekmZEdReXK+Kxueeb+cQPVi6vxAu2CMALAekxoYPV46y6CV1Xj1Zg+0bru1KR2e27A7ggjAKzHhJmRlxdvV9auAwp3BuiJ/ztJfizPAO2GMALAeto5jGwtLNVTmZslSQ9dPEg9O4e2y/MCqEMYAWA97djA6jn07hl3rVdn9O+qCWkJbf6cABoijACwnnbsGZm7ZKdW5hxUuDNAGVcM5SJ4gAkIIwCsp52WaXYVl+vPX2yUJD1w4SDFdwpp0+cD0DjCCADraYcw4vUa+u0HP9a/e+aXp7A8A5iFMALAetohjLz1Q46Wbd+vkEB//fGKk1ieAUxEGAFgPW3cwLrnYKUyFtZtbnbfBQOUGM27ZwAzEUYAWE8bNrAahqEHP1qjcrdHaUmddf3oXq3+HACahzACwHracJnm49V5+nrTPgX5++mPV7K5GWAFhBEA1uL1SoZR9+9WDiMHyt2a9cl6SdIdZ/dV327hrfr4AFqGMALAWg7Pikit3jPy+L82qLjcrQGxEbr1jD6t+tgAWo4wAsBaDveLSK06M/LtliJ9sHK3HA4p48qhCgrg1x9gFYxGANZy5MxIK4WRSrdHD3y0RpI06dQkjUzs3CqPC6B1EEYAWEsbhJFn/rNZOfsrFBcVrHsvGNgqjwmg9RBGAFhLK/eMrMsr0cuLd0iSHh+fonBn+1wJGEDTEUYAWMvhMOJwSH4n9ivK4zX0wEdr5fEaumhonM4ZFNsKBQJobYQRANbSihuevfX9Lq3OPagIZ4AevmTwCT8egLZBGAFgLa204Vmhq0pPfL5JknTvBQMUGxl8opUBaCMtCiOzZ89WcnKygoODlZqaqsWLFx/3/Orqaj344INKSkqS0+lUnz599Oqrr7aoYAA+rpXCyMxP16u0ulbDekbp2lFJrVAYgLbS7NE+f/58TZs2TbNnz9Zpp52mF198UePGjdP69euVmJjY6H0mTJigvXv36pVXXlHfvn1VWFio2iOb1ADgsFYII19vKtS/fsyXn0P6/eVD5c+W74ClOQzj8L7LTTNq1CiNHDlSc+bMqT82aNAgjR8/XhkZGUed//nnn+vqq6/W9u3b1aVLlxYV6XK5FBUVpZKSEkVGRrboMQB0EGvXSkOHSjEx0r59zb57pduj9GcWKXd/pSafnqyHLqZXBDBLU1+/m7VM43a7lZWVpfT09AbH09PTtWTJkkbv8/HHHystLU1PPPGEevToof79++uee+5RZWXlMZ+nurpaLperwQ2ATZxgA+tzX21R7v5KxUUFa/p5/VuxMABtpVmjvaioSB6PR7GxDd8eFxsbq4KCgkbvs337dn377bcKDg7WRx99pKKiIv3mN7/R/v37j9k3kpGRoZkzZzanNAC+4gSWabYWlurvi7dLkh69dIjC2FME6BBa1MDqcDRcfzUM46hjh3m9XjkcDs2bN0+nnHKKLrzwQj399NOaO3fuMWdHZsyYoZKSkvpbbm5uS8oE0BG1MIwYhqGHFqxTjcfQOQO7KX0we4oAHUWzRntMTIz8/f2PmgUpLCw8arbksLi4OPXo0UNRUVH1xwYNGiTDMLR7927169fvqPs4nU45nc7mlAbAVxwOI83cffXj1Xlaur1YwYF+evTSIcf8AwmA9TRrZiQoKEipqanKzMxscDwzM1Njxoxp9D6nnXaa8vLyVFZWVn9s8+bN8vPzU8+ePVtQMgCf1oKeEVdVjR77dIMk6Y6z+ymhS2hbVAagjTR7mWb69Ol6+eWX9eqrr2rDhg266667lJOToylTpkiqW2KZNGlS/fnXXHONoqOjdeONN2r9+vX65ptvdO+99+qmm25SSEhI630nAHxDC5Zpnv5ys4rKqtW7a5huHpvcRoUBaCvN7u6aOHGiiouLNWvWLOXn5yslJUULFy5UUlLdpkL5+fnKycmpPz88PFyZmZm64447lJaWpujoaE2YMEGPP/54630XAHxHM8PI2j0len3pTknSY5elyBlw4hfXA9C+mr3PiBnYZwSwkYULpYsuklJTpRUrjnuqx2voitnfafXuEl06LF7P/nJEOxUJoCnaZJ8RAGhzzWhgnb88V6t3lyjcGaDfXTSojQsD0FYIIwCspYkNrPvL3Xrii42SpOnn9Vc3LoQHdFiEEQDW0sSekT9/sVEHK2o0sHuEJo3mQnhAR0YYAWAtTQgjq3IP6p3ldZshzrosRQH+/CoDOjJGMABr+Zkw4vEaevifa2UY0hUje+iU5JZdgBOAdRBGAFjLzzSwvrM8Rz/uLlGEM0AzxtG0CvgCwggAazlOA+v+cree+HyTJOnu9P7qGsFlIwBfQBgBYC3HWaZ54vONKqmsa1r91ak0rQK+gjACwFqOEUZW5R7U/BV1TauPjadpFfAljGYA1tJIz4j3J02rJ/eiaRXwJYQRANbSSM/Iuyty9eOhnVbvHzfQpMIAtBXCCABr+ckyzcEKt/70ed1Oq9PO7aduEey0CvgawggAa/lJGHk6c7MOVNSof2y4rh/Ty7y6ALQZwggAazkijKzLK9Gby3ZJkmZemqJAmlYBn8TIBmAth8KI4eenh/+5Tl5DumRYvEb3iTa5MABthTACwFoONbBuO1CtrF0HFBrkrwcupGkV8GWEEQDWcmhmZGlOiSTp9rP7Ki4qxMyKALQxwggAazkURspqpd4xYZp8erLJBQFoa4QRAJZy0FUhSar189fDlwyWM6DxC+YB8B2EEQCWYRiGlm/bJ0nq1S1CZw7oZnJFANrD0VeiAgCTfLa2QAcP1M2M/GJwnMnVAGgvzIwAsIRKt0ePf7pe/t66d9NERdC0CtgFYQSAJcz+eqvySqoUeXi+NoCJW8AuCCMATLeruFwvLtouSRoWF1530J/GVcAuCCMATPfYp+vl9ng1tl+M4iKC6g4yMwLYBmEEgKn+u7FQ/95QqAA/hx65ZIgcP7lQHgDfRxgBYJrqWo9mfrJOknTT6cnq2y38qKv2AvB9hBEApnnl2x3aWVyhrhFO3XF237qDhBHAdggjAEyRX1Kp5/6zVZL0wIUDFREcWPeFw2GEBlbANggjAEzxh4UbVVnjUVpSZ40f3uN/Xzh01V5mRgD7IIwAaHfLthfrk9V58nNIMy8bIofD8b8vskwD2A5hBEC7qvF49cg/65pWrxmVqCHxUQ1PIIwAtkMYAdCu3li6S5v2lqpzaKDuSR9w9AmEEcB2CCMA2s2+0mr9JXOzJOm+CwaqU2jQ0SfRwArYDmEEQLv542cbVVpdq5N6RmlCWkLjJ9HACtgOYQRAu8jatV8frNwtSZp1WYr8/RyNn8gyDWA7hBEAbc7jNfTwoabViWkJGp7Q6dgnE0YA2yGMAGhzb/2Qo3V5LkUGB+i+CxppWj0SPSOA7RBGALSp/eVuPfnFJknS3ekDFB3uPP4d6BkBbIcwAqBNPfH5RpVU1mhQXKSuHZX483dgmQawHcIIgDazMueA3lmeK0l67LIhCvBvwq8cwghgO4QRAG2irml1rSTp/1J7Kq1Xl6bdkTAC2A5hBECbeOuHHK3d41JEcIDuHzew6XekgRWwHcIIgFZXXFZd37R6T/oAxfxc0+qRaGAFbIcwAqDVPfH5JpVU1mhwU5tWj8QyDWA7hBEArWplzgHNX3GoaXV8E5tWj0QYAWyHMAKg1dR6vHpowf+aVlOTmti02uBBCCOA3RBGALSaN5bt0ro8l6JCAjWjOU2rhxnG/3pGaGAFbIMwAqBVFLqq9NSXmyVJ913QhJ1WG+P1/u/fzIwAtkEYAdAqHv/XBpVV12pYQif98uRmNq0edniJRiKMADbSojAye/ZsJScnKzg4WKmpqVq8eHGT7vfdd98pICBAw4cPb8nTArCo77YW6ePVefJzSL8fnyI/P0fLHogwAthSs8PI/PnzNW3aND344IPKzs7W2LFjNW7cOOXk5Bz3fiUlJZo0aZLOOeecFhcLwHqqaz166NBOq5NG91JKj6iWP9iRYYSeEcA2mh1Gnn76aU2ePFk333yzBg0apGeeeUYJCQmaM2fOce9366236pprrtHo0aNbXCwA63l58Q5t31eumHCnpqf3P7EHO9y8KjEzAthIs8KI2+1WVlaW0tPTGxxPT0/XkiVLjnm/1157Tdu2bdMjjzzSpOeprq6Wy+VqcANgPTnFFXruqy2SpIcuHqTI4MATe0BmRgBbalYYKSoqksfjUWxsbIPjsbGxKigoaPQ+W7Zs0f3336958+YpoIl/6WRkZCgqKqr+lpCQ0JwyAbQDwzD00D/XqqrGqzF9onXpsPgTf9Ajr0vjaGHfCYAOp0UNrI6f/JIwDOOoY5Lk8Xh0zTXXaObMmerfv+nTtzNmzFBJSUn9LTc3tyVlAmhD/1qTr0Wb9ynI30+PjU9p9HdAs7HhGWBLzRrxMTEx8vf3P2oWpLCw8KjZEkkqLS3VihUrlJ2drdtvv12S5PV6ZRiGAgIC9OWXX+rss88+6n5Op1NOZwv2KADQLlxVNZr5yXpJ0q/P7KM+XcNb54G5Yi9gS82aGQkKClJqaqoyMzMbHM/MzNSYMWOOOj8yMlJr1qzRqlWr6m9TpkzRgAEDtGrVKo0aNerEqgdgiie/2KR9pdXqHROmX5/Zp/UemCv2ArbU7BE/ffp0XXfddUpLS9Po0aP10ksvKScnR1OmTJFUt8SyZ88evf766/Lz81NKSkqD+3fr1k3BwcFHHQfQMazKPag3lu2SJD1+eYqCA1txFoNlGsCWmj3iJ06cqOLiYs2aNUv5+flKSUnRwoULlZSUJEnKz8//2T1HAHRMtR6vHvhwjQxDumJED43pE9PKT0AYAezIYRiGYXYRP8flcikqKkolJSWKjIw0uxzAtl5evF2P/2uDokIC9Z+7z1BMS64/czzZ2dLIkVJ8vLRnT+s+NoB219TXb65NA6BJcvdX1F8Ib8a4ga0fRCQaWAGbIowA+FmGYejBBWtVWePRqOQumpDWRnv/0MAK2BJhBMDPWrBqj77ZvE9BAX7KuGJoyy+E93PoGQFsiTAC4Lj2l7v12KcbJElTz+6r3q21p0hjCCOALRFGABzX45+u1/5ytwbERuj//aIV9xRpDD0jgC0RRgAc0zeb9+nD7D1yOKQ/XjlUQQFt/CuDnhHAlggjABpV4a7VgwvWSJKuH91LIxI7t/2TskwD2BJhBECjnvxis3L3Vyo+Klj3nD+gfZ6UMALYEmEEwFGydu3Xa0t2SJJ+f8VQhTvbKRwQRgBbIowAaKCqxqN73/9RhiFdObKnzhrQrf2enAZWwJYIIwAaePY/W7R9X7m6Rjj10MWD2vfJaWAFbIkwAqDemt0levGb7ZKkx8enqFNoUPsWwDINYEuEEQCSJHetV/e+v1oer6GLT4rT+UO6t38RhBHAlggjACRJs7/eqo0FpeoSFqSZlw4xpwjCCGBLhBEAWrunRM9/tVWS9MglgxXdFlfkbQoaWAFbIowANldd69Hd765WrdfQuJTuunRYvHnF0MAK2BJhBLC5v/57izbtLVV0WJAeH58ih6ONrsjbFCzTALZEGAFsbGXOAb2waJsk6Q9XDDVveeYwwghgS4QRwKYq3R7d8+5qeQ3pihE9zHn3zE/RMwLYEmEEsKknvtio7UXlio106pFLTHr3zE/RMwLYEmEEsKEl24r02nc7JUl/uvIkRYUGmlvQYSzTALZEGAFspqSiRne/u1qS9MtTEnVme1575ucQRgBbIowANmIYhh5csEb5JVVKjglr/2vP/BzCCGBLhBHARhas2qNPf8yXv59Df5k4XKFBFnvRp4EVsCXCCGATufsr9PCCdZKkaef00/CETuYW1BgaWAFbIowANuDxGrr73dUqra5ValJn/frMPmaX1DiWaQBbIowANvDCom36Yed+hTsD9MzE4Qrwt+jQJ4wAtmTR30gAWkvWrgN6OnOzJOnRS4cooUuoyRUdB2EEsCXCCODDSiprNPXtbHm8hi4ZFq8rR/Ywu6Tjo4EVsCXCCOCjDMPQ/R/8qD0HK5XYJVR/uNzki+A1BQ2sgC0RRgAfNe/7HH22tkCB/g49f80IRQRbZJfV42GZBrAlwgjggzbkuzTr0/WSpN9eMFAn9exkbkFNRRgBbIkwAviYCnetbn9rpdy1Xp01oKtuOi3Z7JKajp4RwJYII4APMQxDv1uwVtv21V2N98mrhsnPz+J9IkeiZwSwJcII4EPe/iFXH67cI38/h/569QhFhzvNLql5WKYBbIkwAviINbtL9OjHddu933v+AJ3aO9rkilqAMALYEmEE8AEHK9z69bwsuT1enTc4Vrf+orfZJbUMYQSwJcII0MF5vYamv7tauw9UKik6VE9eNcz6+4kcCw2sgC0RRoAObs6ibfpqY6GcAX6afe1IRYV0gP1EjoUGVsCWCCNAB/b1pkI9+eUmSdJjl6VoSHyUyRWdIJZpAFsijAAd1PZ9Zbrj7WwZhvTLUxI14eQEs0s6cYQRwJYII0AHVFpVo1teX6HSqlqlJXXWzEuHmF1S6yCMALZEGAE6GK/X0F3zV2nbvnJ1jwzW7F+NVFCAjwxlGlgBW/KR32CAffzl35v17w11DasvTUpVt4hgs0tqPTSwArZEGAE6kH/9mK/nvtoqSfrjlUM7zgXwmoplGsCWCCNAB5Gdc0DT310lSbr59GRdPqKnuQW1BcIIYEuEEaADyN1foVteX6HqWq/OHthNMy4cZHZJbYOeEcCWCCOAxbmqajT5H8tVVObWoLhIPfvLEfLvSFfibQ56RgBbIowAFlbj8eq2eSu1eW+ZYiOdevWGNIU7ffiFmmUawJYII4BFGYahRz9ep8VbihQS6K9Xrj9ZcVEhZpfVtggjgC0RRgCLmv31Ns37PkcOh/TXq4crpUcH3+q9KQgjgC21KIzMnj1bycnJCg4OVmpqqhYvXnzMcz/88EOdd9556tq1qyIjIzV69Gh98cUXLS4YsIN3fsjRn7+ou+bMIxcPVvqQ7iZX1E5oYAVsqdlhZP78+Zo2bZoefPBBZWdna+zYsRo3bpxycnIaPf+bb77Reeedp4ULFyorK0tnnXWWLrnkEmVnZ59w8YAvyly/Vw98tEaS9Jsz++iG05JNrqgd0cAK2JLDMAyjOXcYNWqURo4cqTlz5tQfGzRokMaPH6+MjIwmPcaQIUM0ceJEPfzww0063+VyKSoqSiUlJYqMjGxOuUCHsmLnfl378veqrvVqQlpP/enKk+Rw+Og7Zxrj7y95vVJenhQXZ3Y1AE5QU1+/mzUz4na7lZWVpfT09AbH09PTtWTJkiY9htfrVWlpqbp06XLMc6qrq+VyuRrcAF+3eW+pbpq7XNW1Xp0zsJv+cPlQewURr7fuJjEzAthMs8JIUVGRPB6PYmNjGxyPjY1VQUFBkx7jqaeeUnl5uSZMmHDMczIyMhQVFVV/S0jwgUujA8exo6hc1778vVxVtRqZ2EnPXzNSAf426y8/vEQjEUYAm2nRb7uf/rVmGEaT/oJ7++239eijj2r+/Pnq1q3bMc+bMWOGSkpK6m+5ubktKRPoEHYfqNC1f1+mfaXVGtg9Qq/ecLJCgmzYwHm4eVWigRWwmWb9+RETEyN/f/+jZkEKCwuPmi35qfnz52vy5Ml67733dO655x73XKfTKafT2ZzSgA5pr6tK1/z9e+WVVKl31zC9MXmUOoUGmV2WOZgZAWyrWTMjQUFBSk1NVWZmZoPjmZmZGjNmzDHv9/bbb+uGG27QW2+9pYsuuqhllQI+pqisWtf8fZly9lcosUuo3rr5VHWNsHEIP3JmhDAC2EqzR/z06dN13XXXKS0tTaNHj9ZLL72knJwcTZkyRVLdEsuePXv0+uuvS6oLIpMmTdJf//pXnXrqqfWzKiEhIYqKssEmTkAjDpS7dd0rP2jbvnLFRwVr3s2j1D0q2OyyzEUYAWyr2SN+4sSJKi4u1qxZs5Sfn6+UlBQtXLhQSUlJkqT8/PwGe468+OKLqq2t1W233abbbrut/vj111+vuXPnnvh3AHQwRWXV+tXL32tjQam6Rjg175ZTldAl1OyyzHdkGPGzWfMuYHPN3mfEDOwzAl9R6KrSNS9/r62FZeoW4dRbt5yqvt3CzS7LGvbskXr2rJsVqakxuxoAraCpr9/MhQLtpKCkStf8fZm2F5UrLipYb91yqpJjwswuyzq4Lg1gW4x6oB3sOVipa/6+TLuKK9SjU4jevuVUJUazNNMAYQSwLUY90Ma2FpZp0it1b99N7BKqt24ZpZ6dCSJHIYwAtsWoB9rQqtyDuvG1H3Sgoka9u4Zp3s2jFBcVYnZZ1sQVewHbIowAbeSbzfs05c0sVbg9GtYzSq/deIq6hNl0Q7Om4Iq9gG0x6oE28PHqPN397irVeAyN7RejF36VqjAnw+24WKYBbItRD7QiwzD08uId+sNnG2QY0sUnxempCcPkDGDp4WcRRgDbYtQDraTG49UjH6/TW9/Xbfo3aXSSHrlkiPz9fv4ikhBhBLAxRj3QClxVNbpt3kot3lIkh0P63UWDddNpvZp0NWscQgMrYFuEEeAE5e6v0E1zl2tLYZlCAv317C9H6LzBx7+KNRpBAytgW4x64AQs2Vqk29/O1v5yt2IjnXrl+pOV0oMLQLYIyzSAbTHqgRY43Kia8dkGeQ0ppUek/j4pjT1ETgRhBLAtRj3QTBXuWv32gzX6ZHWeJOmKkT30h8uHKjiQXocTQs8IYFuEEaAZdhSV69dvZmljQakC/Bx66OLBmjQ6iUbV1kDPCGBbjHqgiRZk79GDH61RudujmHCnZl87UqckdzG7LN/BMg1gW4x64GdUuGv1yD/X6b2s3ZKkUcld9OwvRyg2MtjkynwMYQSwLUY9cBwbC1y6/a1sbS0sk59DuuPsfpp6Tj82MmsLhBHAthj1QCM8XkMvL96up77cLLfHq9hIp56ZOEKj+0SbXZrvooEVsC3CCPATu4rLdc97q7V85wFJ0tkDu+nP/3eSosOdJlfm42hgBWyLUQ8cYhiG5n2foz8s3KAKt0dhQf566OLBmnhyAu+WaQ8s0wC2xagHVPeW3Qc/WqMl24ol1TWpPnnVMCV0CTW5MhshjAC2xaiHrblrvXrpm2169qutctd6FRzop3vPH6gbx/SSH02q7YswAtgWox62lbXrgGZ8+KM27y2TJI3tF6Pfjx+qxGhmQ0xxuGeEBlbAdggjsJ1CV5X+9PkmfbCybt+QLmFBevjiwbpseDy9IWZiZgSwLUY9bMNd69Vr3+3Qs//ZonJ33V/hV6X21AMXDlLnsCCTqwNhBLAvRj18nmEYyly/VxmfbdSOonJJ0rCETnr0ksEakdjZ5OpQjzAC2BajHj5t+c79+uNnG5W1q27PkJhwp+4fN1BXjOhBg6rVsOkZYFuEEfikzXtL9cTnG/XvDYWSpOBAP910WrJ+fWYfRQQHmlwdGsWmZ4BtMerhUzbvLdWz/9mif63Jl2FI/n4OTTw5QXee048L21kdyzSAbTHq4RM25Lv03FdbtHBNQf2xcSnddc/5A9Sna7iJlaHJCCOAbTHq0WEZhqEVuw7oxUXb9e8Ne+uPj0vprqnn9NOguEgTq0OzEUYA22LUo8PxeA19sa5AL32zXatyD0qSHA7pwqFxmnp2Pw3oHmFugWgZGlgB2yKMoMM4WOHWeyt2641lu5Szv0KSFBTgpytH9tDk03urbzeWYzo0GlgB22LUw/LW7C7R60t36uPVeaqu9UqSOoUGatKpSbpudC91jXCaXCFaBcs0gG0x6mFJJZU1+mR1nt5bkavVu0vqjw+Ki9Sk0Um6bHi8QoP47+tTCCOAbTHqYRker6Fl24v13opcfba2oH4WJNDfoXEpcZo0OkmpSZ25foyvIowAtsWoh6kMw9CaPSX656o8ffpjnva6quu/1j82XBPSEjR+RA/FhLMU4/O4ai9gW4QRtDvDMLQuz6Uv1hXo0x/z668XI0mRwQG6ZFi8JqQl6KSeUcyC2AkzI4BtMerRLjxeQ9k5B/T52gJ9vq5Auw9U1n8tONBP5w6K1aXD4nXGgK5yBvCXsS0RRgDbYtSjzRyscOubLUX678ZCLdq8T/vL3fVfCw700xn9u2pcSpzOGxyrMCf/FW2PMALYFqMerabG49Xq3IP6dmuRvt1SpJU5B+Q1/vf1iOAAnTsoVucP6a4z+ndVSBAzIDgCm54BtkUYQYvVerxan+/SDzv2a+m2Yi3bXqxyt6fBOQNiI3TmwK46e0A3pSZ1VoC/n0nVwvLY9AywLUY9mqy8ulardx9Uds5Bfb9jv7J27j8qfHQODdSYvjEa2zdGp/eLUc/OoSZViw6HZRrAthj1aFStx6sthWVas6dEq3MPamXOQW0qcDVYdpHqll5O6dVFpyR30Wl9YzQ4LlJ+frwDBi1AGAFsi1EPVdV4tKmgVBvyXVqf79KaPSXakO9SVY33qHPjo4I1Iqmz0pI6a1RytAZ0j5A/4QOtgTAC2Baj3kZqPF7tKi7X5r1l2ry3VFv2lmljgUs7isqPmvGQpHBngFJ6RGpojyiNTOysEYmd1T0quP0Lhz3QwArYFmHExxiGocLSau0sKtfO4nJt31eubfvKtb2oTDnFFaptLHVIig4L0qC4SA2Ki1BKjyil9IhScnQYSy5oPzSwArbFqO9gDMOQq7JWew5WaveBCu0+UKndByqVe6BCOcUV2rW/vNHllcPCgvzVNzZC/buFq19suPrHRmhwXKS6RjjZ7RTmYpkGsC1GvYV4vIb2l7u111WlwtIqFZRUa6+rSgUlVcorqVR+SZXyD1Ye9Q6Wn/L3cyi+U7B6RYepT9dw9e4apt4xdR/jooIJHbAmwghgW4z6NmQYhkqra3WwvEb7K9w6UO5WcblbxWXVKi53q6isWsVlbu0rrda+smoVl1U32rvRmC5hQUroHKKenUPVs3OIenYOUWJ0mJK6hKpH5xAFsp8HOhp6RgDbalEYmT17tv785z8rPz9fQ4YM0TPPPKOxY8ce8/xFixZp+vTpWrduneLj43XfffdpypQpLS66vRiGoepar8qqa1VWVavSqlqVVtXIdcRHV2WNXFU1clXWqqTSrYMVNTpYWaODFTUqqXSrxtPEdHGIwyFFhznVPcqp2IhgxUYFKzYiWHGdghUfFaL4TsGKiwph91L4HnpGANtq9qifP3++pk2bptmzZ+u0007Tiy++qHHjxmn9+vVKTEw86vwdO3bowgsv1C233KI333xT3333nX7zm9+oa9euuvLKK1vlm2ipN5ft0spdB1TurlWF26Oy6lpVVNd9LHfXBZBjNXw2R0igv7qEBalTaKCiw52KCQtSdHiQuoQ5FR0epK4RTnUNd6pbhFNdwoLYpRT2xDINYFsOwzCa9Wo7atQojRw5UnPmzKk/NmjQII0fP14ZGRlHnf/b3/5WH3/8sTZs2FB/bMqUKVq9erWWLl3apOd0uVyKiopSSUmJIiMjm1PucU19O1sfr85r0rlhQf6KCA5UeHCAIoIDFBEcqKiQQEUGBygyJFCRwYHqFBqoTiGBigoNVKeQuvDROTSIWQygKXr2lPbskbKypJEjza4GQCto6ut3s/4EcbvdysrK0v3339/geHp6upYsWdLofZYuXar09PQGx84//3y98sorqqmpUWBg4FH3qa6uVnV1dYNvpi38v53f6pZtKxXg51Cgv58C/R0KOPQx0N9PQf5+9Z/7VTmktikDgCQdOFD3kZkRwHaaNeqLiork8XgUGxvb4HhsbKwKCgoavU9BQUGj59fW1qqoqEhxcXFH3ScjI0MzZ85sTmktkrJmqfT+223+PACaoVMnsysA0M5a9CfIT98aahjGcd8u2tj5jR0/bMaMGZo+fXr95y6XSwkJCS0p9fguu0xKTm79xwXQMikpUiO9ZwB8W7PCSExMjPz9/Y+aBSksLDxq9uOw7t27N3p+QECAoqOjG72P0+mU0+lsTmktM3Fi3Q0AAJimWW/bCAoKUmpqqjIzMxscz8zM1JgxYxq9z+jRo486/8svv1RaWlqj/SIAAMBemv0e0unTp+vll1/Wq6++qg0bNuiuu+5STk5O/b4hM2bM0KRJk+rPnzJlinbt2qXp06drw4YNevXVV/XKK6/onnvuab3vAgAAdFjN7hmZOHGiiouLNWvWLOXn5yslJUULFy5UUlKSJCk/P185OTn15ycnJ2vhwoW666679Le//U3x8fF69tlnTd9jBAAAWEOz9xkxQ1vtMwIAANpOU1+/2eoTAACYijACAABMRRgBAACmIowAAABTEUYAAICpCCMAAMBUhBEAAGAqwggAADAVYQQAAJiKMAIAAExFGAEAAKYijAAAAFMRRgAAgKkIIwAAwFSEEQAAYCrCCAAAMBVhBAAAmIowAgAATEUYAQAApiKMAAAAUxFGAACAqQgjAADAVIQRAABgqgCzC2gKwzAkSS6Xy+RKAABAUx1+3T78On4sHSKMlJaWSpISEhJMrgQAADRXaWmpoqKijvl1h/FzccUCvF6v8vLyFBERIYfDYXY5pnO5XEpISFBubq4iIyPNLsen8bNuP/ys2w8/6/Zj95+1YRgqLS1VfHy8/PyO3RnSIWZG/Pz81LNnT7PLsJzIyEhb/uc2Az/r9sPPuv3ws24/dv5ZH29G5DAaWAEAgKkIIwAAwFSEkQ7I6XTqkUcekdPpNLsUn8fPuv3ws24//KzbDz/rpukQDawAAMB3MTMCAABMRRgBAACmIowAAABTEUYAAICpCCM+orq6WsOHD5fD4dCqVavMLsfn7Ny5U5MnT1ZycrJCQkLUp08fPfLII3K73WaX5jNmz56t5ORkBQcHKzU1VYsXLza7JJ+TkZGhk08+WREREerWrZvGjx+vTZs2mV2WLWRkZMjhcGjatGlml2JJhBEfcd999yk+Pt7sMnzWxo0b5fV69eKLL2rdunX6y1/+ohdeeEEPPPCA2aX5hPnz52vatGl68MEHlZ2drbFjx2rcuHHKyckxuzSfsmjRIt12221atmyZMjMzVVtbq/T0dJWXl5tdmk9bvny5XnrpJZ100klml2JZvLXXB3z22WeaPn26PvjgAw0ZMkTZ2dkaPny42WX5vD//+c+aM2eOtm/fbnYpHd6oUaM0cuRIzZkzp/7YoEGDNH78eGVkZJhYmW/bt2+funXrpkWLFukXv/iF2eX4pLKyMo0cOVKzZ8/W448/ruHDh+uZZ54xuyzLYWakg9u7d69uueUWvfHGGwoNDTW7HFspKSlRly5dzC6jw3O73crKylJ6enqD4+np6VqyZIlJVdlDSUmJJPH/uA3ddtttuuiii3TuueeaXYqldYgL5aFxhmHohhtu0JQpU5SWlqadO3eaXZJtbNu2Tc8995yeeuops0vp8IqKiuTxeBQbG9vgeGxsrAoKCkyqyvcZhqHp06fr9NNPV0pKitnl+KR33nlHWVlZWrFihdmlWB4zIxb06KOPyuFwHPe2YsUKPffcc3K5XJoxY4bZJXdYTf1ZHykvL08XXHCBrrrqKt18880mVe57HA5Hg88NwzjqGFrP7bffrh9//FFvv/222aX4pNzcXN15552aN2+egoODzS7H8ugZsaCioiIVFRUd95xevXrp6quv1ieffNLgF7bH45G/v7+uvfZa/eMf/2jrUju8pv6sD/8yycvL01lnnaVRo0Zp7ty58vMjz58ot9ut0NBQvffee7r88svrj995551atWqVFi1aZGJ1vumOO+7QggUL9M033yg5OdnscnzSggULdPnll8vf37/+mMfjkcPhkJ+fn6qrqxt8ze4IIx1YTk6OXC5X/ed5eXk6//zz9f7772vUqFHq2bOnidX5nj179uiss85Samqq3nzzTX6RtKJRo0YpNTVVs2fPrj82ePBgXXbZZTSwtiLDMHTHHXfoo48+0tdff61+/fqZXZLPKi0t1a5duxocu/HGGzVw4ED99re/ZWnsJ+gZ6cASExMbfB4eHi5J6tOnD0GkleXl5enMM89UYmKinnzySe3bt6/+a927dzexMt8wffp0XXfddUpLS9Po0aP10ksvKScnR1OmTDG7NJ9y22236a233tI///lPRURE1PfkREVFKSQkxOTqfEtERMRRgSMsLEzR0dEEkUYQRoAm+PLLL7V161Zt3br1qKDH5OKJmzhxooqLizVr1izl5+crJSVFCxcuVFJSktml+ZTDb50+88wzGxx/7bXXdMMNN7R/QcAhLNMAAABT0X0HAABMRRgBAACmIowAAABTEUYAAICpCCMAAMBUhBEAAGAqwggAADAVYQQAAJiKMAIAAExFGAEAAKYijAAAAFMRRgAAgKn+P2Z4nUSWIYs0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# compare sigmoid and step\n",
    "x = np.arange(-5.0, 5.0, 0.1)\n",
    "y1 = sigmoid(x)\n",
    "y2 = step_function(x)\n",
    "y3 = softmax(x)\n",
    "\n",
    "plt.plot(x, y1)\n",
    "plt.plot(x, y2, 'r')\n",
    "# plt.plot(x, y3, 'g')\n",
    "plt.ylim(-0.1, 1.1) #指定图中绘制的y轴的范围\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbb89a00",
   "metadata": {},
   "source": [
    "### ReLU函数\n",
    "\n",
    "ReLU(Rectified Linear Unit)函数，也就是调整过的线性单元，是比较常用的激活函数，其表达式为：\n",
    "$$\n",
    "h(x)=\n",
    "\\begin{cases}\n",
    "0, &if\\ \\ x \\le 0\\\\[2ex]\n",
    "x, &if\\ \\ x \\gt 0\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "在零点好像不可导？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "4fc0b145",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ReLU\n",
    "def relu(x):\n",
    "    return np.maximum(0, x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "0f5ffdfb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAGdCAYAAAAvwBgXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAon0lEQVR4nO3dd3hUdd7+8XuSwNCS0JsJELFQQkuiCDZQRJFVwApCLD8VcQHBrKLorgKrG3etP1ECuD660m2groiCCqg8qAm9Sk8gBAhIJgkwaef5Y0IIAak5853yfl3X/HFOzjXfjyckc3vmnhOHZVmWAAAADAgxPQAAAAheBBEAAGAMQQQAABhDEAEAAMYQRAAAgDEEEQAAYAxBBAAAGEMQAQAAxoSZHuBUSkpKlJmZqfDwcDkcDtPjAACAM2BZlnJzc9W0aVOFhJz6modPB5HMzExFR0ebHgMAAJyDjIwMRUVFnfIYnw4i4eHhkjz/IREREYanAQAAZ8Llcik6OrrsdfxUfDqIHH07JiIigiACAICfOZNaBWVVAABgDEEEAAAYQxABAADGEEQAAIAxBBEAAGAMQQQAABhDEAEAAMYQRAAAgDEEEQAAYAxBBAAAGEMQAQAAxhBEAACAMQQRAABgDEEEAAAYQxABAADGEEQAAIAxBBEAAGAMQQQAABhDEAEAAMYQRAAAgDEEEQAAYIytQWTMmDFyOBzHPRo3bmznkgAAwI+E2b1A27ZttWDBgrLt0NBQu5cEAAB+wvYgEhYWxlUQAABwUrZ3RDZt2qSmTZsqJiZG/fv319atW//wWLfbLZfLddwDAAAELluDSOfOnfXBBx/o66+/1jvvvKOsrCx17dpV+/fvP+nxycnJioyMLHtER0fbOR4AADDMYVmW5a3F8vPz1bJlS40aNUpJSUknfN3tdsvtdpdtu1wuRUdHKycnRxEREd4aEwAAnAeXy6XIyMgzev22vSNSXs2aNdWuXTtt2rTppF93Op1yOp3eHAkAABjk1fuIuN1urV+/Xk2aNPHmsgAAwEfZGkSeeOIJLVq0SNu2bdPPP/+sO+64Qy6XS/fdd5+dywIAgNOwLEtTlu5QnrvI6By2BpGdO3dqwIABuvTSS3XbbbepatWqWrp0qZo3b27nsgAA4DT+s2S7/jZnje5IWaKCohJjc9jaEZk5c6adTw8AAM7B0q379fcv10uS7oiPUtUwc3/xhb81AwBAENl18LCGTlum4hJLfTs21YNXxRidhyACAECQOFJYrCFT0rQ/v0BtmkQo+bb2cjgcRmciiAAAEAQsy9Izs1dr9a4c1alRRZMS41W9qvm//0YQAQAgCLz303Z9umyXQkMcevueOEXXrWF6JEkEEQAAAt6SLdl6ca6nnDq6Vyt1vai+4YmOIYgAABDAdv5+SMOmL/eZcmpFBBEAAALUkcJiDZmapgP5BYq9IEIv3W6+nFoRQQQAgABkWZZGf7paa3a5VLdmVU1KTFC1KubLqRURRAAACED/89N2zV5+rJx6Qe3qpkc6KYIIAAABZsmWbP2jtJz67M2t1aVlPcMT/TGCCAAAAaR8OfW2ThfogStbmB7plAgiAAAEiMMFxXpkiqec2u6CSP3jtnY+V06tiCACAEAA8JRTV2ltpkv1albVxMR4nyynVkQQAQAgALz74zbNWZGp0BCH3vLhcmpFBBEAAPzcks3ZSv5qgyTpr719u5xaEUEEAAA/lnHgkIZOX+Ypp8ZdoPu7tjA90lkhiAAA4KeOllN/P1ToKaf28/1yakUEEQAA/JBlWXr601Vat9tTTp3kJ+XUiggiAAD4oXd/3KbPVmQqLMShtwfGqamflFMrIogAAOBnftp87M6pf+3dWldc6D/l1IoIIgAA+JGMA4c0bPoylVjS7XFRus/PyqkVEUQAAPAThwuKNbi0nNo+KlIv9ov1u3JqRQQRAAD8gGVZGvXJKq3f7VL9WlU1cZB/llMrIogAAOAH3vlhq75YWVpOvcd/y6kVEUQAAPBxP2zap5dK75z6tz+1UWc/LqdWRBABAMCHZRw4pOEzlqvEku6Ij9K9XZqbHqlSEUQAAPBRhwqK9PAHqTp4qFAdoiL1Ql//L6dWRBABAMAHWZalUR+v0oasXE851U/vnHo6BBEAAHzQ5MVb9d9VuxUW4tCEgfFqEhkY5dSKCCIAAPiYHzbt0z/necqpz9/SRpfH1DU8kX0IIgAA+JD0/Yc0bLqnnHpXQpQGXRFY5dSKCCIAAPiIQwVFGjwlVTmHC9UhurbG9Qm8cmpFBBEAAHzA8eVUpyYFyJ1TT4cgAgCAD5hUrpyaMihOjSOrmR7JKwgiAAAYtvi3ffrX0XLqrW11WYvALadWRBABAMCgHfvzy+6cendCtAZ1bmZ6JK8iiAAAYEi+u0iPTElTzuFCdYyurXF92wZ8ObUigggAAAaUL6c2CHdq4qB4OcMCv5xaEUEEAAADUhZt0Zerd6tKqEMpA4OnnFoRQQQAAC9buHGvXv56oyTp+VvaKiGIyqkVEUQAAPCi7dn5emzGclmWNODyaA0MsnJqRQQRAAC8JN/tuXOq60iR4prV1phbg6+cWhFBBAAAL7AsS09+vFK/7clTg3CnUoK0nFoRQQQAAC+YsHCL5q7OUpVQhyYOilOjiOAsp1ZEEAEAwGbfb9yrV77xlFPH3hqr+ObBW06tiCACAICNtmfna0RZObWZ7gnycmpFXgsiycnJcjgcGjlypLeWBADAqLwTyqltTI/kc7wSRH799VdNnjxZ7du398ZyAAAYZ1mWnvzIU05tGMR3Tj0d24NIXl6eBg4cqHfeeUd16tSxezkAAHzChIVb9NUaTzk1ZVC8GlJOPSnbg8jQoUPVu3dv9ejR47THut1uuVyu4x4AAPib8uXUcX1iFd+c/xH/I2F2PvnMmTOVlpam1NTUMzo+OTlZY8eOtXMkAABsta3cnVPv6dxMAy6nnHoqtl0RycjI0IgRIzRt2jRVq3Zml6NGjx6tnJycskdGRoZd4wEAUOny3EUa/EGqco8UKb55HY25pa3pkXyebVdE0tLStHfvXsXHx5ftKy4u1uLFi/XWW2/J7XYrNPT40o7T6ZTT6bRrJAAAbGNZlp74cKU27c1TowinUgbGqWoYd8k4HduCyPXXX6/Vq1cft++BBx5Qq1at9NRTT50QQgAA8Gdvf79Z89ZmqWpoCOXUs2BbEAkPD1dsbOxx+2rWrKl69eqdsB8AAH/23YY9enX+b5KkcX3aKq4Z5dQzxTUjAADOw9Z9eRoxc0VZObU/5dSzYuunZipauHChN5cDAMBWuUcKNXhKGuXU88AVEQAAzkFJiaW/fLhSmymnnhfOGAAA5+Ct7zfrm3V7KKeeJ4IIAABn6dv1e/T6Ak859e99KaeeD4IIAABnYeu+PI0sLacOuqKZ7r6Mcur5IIgAAHCGco8U6uEPUpXrLlJC8zp67k+UU88XQQQAgDNQUmIp6cOV2rIvX40jqmnCIMqplYEzCADAGRj/3WbNLyunxqlhOOXUykAQAQDgNBasO1ZOfaFvrDpRTq00BBEAAE5hy748PT5rhSQp8YrmuuuyaLMDBRiCCAAAfyD3SKEGl5ZTL2tRR3/7UxvTIwUcgggAACdRUmLp8VnlyqkD4ymn2oAzCgDASbz53SYtWL9HVcNCNDExXg3CnaZHCkgEEQAAKpi/bo/eWLBJkqec2jG6ttmBAhhBBACAcjbvPVZOvbdLc92VQDnVTgQRAABKuUrLqXnuIl3eoi7lVC8giAAAoNI7p85aoa3Z+WoSWU1vD4xTlVBeJu3GGQYAQNIb327SgvV7PeXUQZRTvYUgAgAIel+vzdKb33rKqS/2jVUHyqleQxABAAS1zXtzlVRaTr2/awvdSTnVqwgiAICg5Smnpim/oFidY+rq2d6tTY8UdAgiAICgVFJi6fGZnnJqU8qpxnDGAQBB6Y0Fv+nbDXvL7pxavxblVBMIIgCAoDNvTZbe/G6zJCm5Xzu1j6ptdqAgRhABAASVTXty9ZcPV0jylFNvj48yO1CQI4gAAIJGzuFCDZ5COdWXEEQAAEGhpMTS47NWaBvlVJ/CdwAAEBReX/CbvtuwV86wEE1KTKCc6iMIIgCAgDdvzW6NP1pOva2d2kVFGp4IRxFEAAABzVNOXSlJeuDKFrotjnKqLyGIAAACVvly6hUX1tUzN1NO9TUEEQBAQCousTRi5nJty87XBbWr6+17KKf6Ir4jAICA9Nr8jVq4cV9pOTVe9Sin+iSCCAAg4Hy1erfe/n6LJOml29sp9gLKqb6KIAIACCgbs3L1l4885dT/d2WM+nWinOrLCCIAgICRc6hQg6ek6lBBsbpcWE/P3NzK9Eg4DYIIACAgFJdYemzmcu3Yf0gX1K6ut+7ppDDKqT6P7xAAICC8+s1GLfptn6pVoZzqTwgiAAC/N3f1bk1Y6Cmn/vP29pRT/QhBBADg1zZm5eqJ0nLqQ1fFqE/HCwxPhLNBEAEA+K3y5dQrL6qnp3tRTvU3BBEAgF8qLrE0vFw5dfyAOMqpfojvGADAL73yzUYtLi2nTr43XnVrVjU9Es4BQQQA4He+XLVbKeXKqW2bUk71VwQRAIBf2ZDl0pMfe8qpD19NOdXfEUQAAH7j4KECDf4grayc+tRNlFP9HUEEAOAXikssDZ+xXOkHDimqTnW9RTk1INj6HUxJSVH79u0VERGhiIgIdenSRV999ZWdSwIAAtTLX2/UD5uyPeXUxATVoZwaEGwNIlFRUXrppZeUmpqq1NRUXXfdderTp4/Wrl1r57IAgADz31WZmrjIU0791x0d1KZphOGJUFkclmVZ3lywbt26evnll/Xggw+e9liXy6XIyEjl5OQoIoJ/dAAQjNbvdum2CUt0uLBYg6+5UM/c3Nr0SDiNs3n9DvPSTCouLtZHH32k/Px8denS5aTHuN1uud3usm2Xy+Wt8QAAPujgoQINnpKqw4XFuuqi+hp146WmR0Ils73ls3r1atWqVUtOp1NDhgzR7Nmz1aZNm5Mem5ycrMjIyLJHdHS03eMBAHxUUXGJhs9YrowDhxVVp7rGD+hEOTUA2f7WTEFBgdLT03Xw4EF98skn+ve//61FixadNIyc7IpIdHQ0b80AQBBKnrtekxZvVbUqIfr00SvphfiRs3lrxusdkR49eqhly5aaNGnSaY+lIwIAwenzlZl6bMZySdL4AZ10S4emhifC2Tib12+vX+OyLOu4qx4AAJS3LtOlUaV3Tn3k2gsJIQHO1rLqM888o169eik6Olq5ubmaOXOmFi5cqHnz5tm5LADAT/2eX6BHpqbqSGGJrr64vkbdyJ1TA52tQWTPnj1KTEzU7t27FRkZqfbt22vevHm64YYb7FwWAOCHypdTm9WtofEDOik0xGF6LNjM1iDy7rvv2vn0AIAA8q+vN+rHzdmqXiVUkxLjVbsGd04NBnwOCgBg3Gcrdmny4q2SpJfvbK/WTfiAQrAgiAAAjFqbmaOnPlklSRpybUv9qT3l1GBCEAEAGHMgv0CPTEnTkcISXXNJAz3JnVODDkEEAGCEp5y6TDt/95RT3+zfkXJqECKIAACMeOmrDfpp835VrxKqyfdSTg1WBBEAgNd9tmKX/v3jNknSK3d2UKvGlFODFUEEAOBVa3blaNTHnnLqo91aqnf7JoYngkkEEQCA1xwtp7qLSnTtJQ30RE/KqcGOIAIA8Iqi4hINm75Muw4eVvN6NfRmf+6cCoIIAMBLkr/aoCVb9qtG1VBNTkxQZI0qpkeCDyCIAABsN3v5Tr1brpx6aeNwwxPBVxBEAAC2WrMrR09/slqS9OduLXVzO8qpOIYgAgCwzf48d1k5tdulDfQXyqmogCACALCFp5y6XLsOHlaLejX0/ymn4iQIIgAAW/xj7gb979b9qlk1VJPvTVBkdcqpOBFBBABQ6T5dtlP/85OnnPrqXR10SSPKqTg5gggAoFKt2ZWj0Z96yqnDul+km2Ipp+KPEUQAAJWmfDm1+6UN9PgNl5geCT6OIAIAqBSFxSUaWnrn1Jj6NfUG5VScAYIIAKBS/GPuei3desBTTk2Mp5yKM0IQAQCct0/Sduq9n7ZLkl69q6MuppyKM0QQAQCcl1U7D2r0bE85dfh1F+mm2MaGJ4I/IYgAAM5Zdmk5taCoRNe1aqjHe1BOxdkhiAAAzklhcYn+PG2ZduccUUz9mnr97o4KoZyKs0QQAQCckxe/XK9ftlFOxfkhiAAAztpHqRl6f8l2SdJrd1NOxbkjiAAAzsqqnQf17Jw1kqTHrrtIN7alnIpzRxABAJyxfbnHyqnXt2qokZRTcZ4IIgCAM3L0zqm7c47owvo19Xp/yqk4fwQRAMAZeeG/6/TLtgOq5QzT5HvjFVGNcirOH0EEAHBaH6Vm6D//u0OS9PrdHXVRQ8qpqBwEEQDAKa3MOFZOHXH9xbqhTSPDEyGQEEQAAH+ofDm1R+tGGnH9xaZHQoAhiAAATqqgqERDpy1TluuILmxQU6/d3YFyKiodQQQAcFIvfLlOv2wvLacmJlBOhS0IIgCAE3yYmqEPjiun1jI8EQIVQQQAcJwVGQf119mecurIHpRTYS+CCACgzN7cIxoyJU0FxSW6oU0jPXYd5VTYiyACAJDkKaf+eaqnnNqyQU29dhflVNiPIAIAkCSN++9ape74XeHOME2+N0HhlFPhBQQRAIBm/ZquqUvTJXnKqS0bUE6FdxBEACDILUv/XX+bs1aS9HiPS9SDciq8iCACAEFsb+4RPTrVU07t2aaRhl93kemREGQIIgAQpI6WU/e43GrZoKZepZwKAwgiABCkxn5BORXmEUQAIAjN/CVd035Ol8MhvdGfcirMsTWIJCcn67LLLlN4eLgaNmyovn37auPGjXYuCQA4jWXpv+u5zzzl1KQel+j61pRTYY6tQWTRokUaOnSoli5dqvnz56uoqEg9e/ZUfn6+ncsCAP7AXtexO6fe2LaRhnannAqzwux88nnz5h23/d5776lhw4ZKS0vTNddcY+fSAIAKCopK9Oi0Zdqb69bFDWvp1bs6Uk6FcbYGkYpycnIkSXXr1j3p191ut9xud9m2y+XyylwAEAzGfLFWaTt+V3g1Tzm1ltOrLwHASXmtrGpZlpKSknTVVVcpNjb2pMckJycrMjKy7BEdHe2t8QAgoM34JV3TS8upb/bvpJj6NU2PBEjyYhAZNmyYVq1apRkzZvzhMaNHj1ZOTk7ZIyMjw1vjAUDAStvxu577bI0k6S83XKLurRoangg4xivX5YYPH67PP/9cixcvVlRU1B8e53Q65XQ6vTESAASFPS7PnVMLiy3d1LYx5VT4HFuDiGVZGj58uGbPnq2FCxcqJibGzuUAAOW4i4r16NS0snLqK3d1kMNBORW+xdYgMnToUE2fPl2fffaZwsPDlZWVJUmKjIxU9erV7VwaAILemM/XaVn6Qcqp8Gm2dkRSUlKUk5Ojbt26qUmTJmWPWbNm2bksAAS96T+na8YvpeXUAZRT4btsf2sGAOBdaTsO6PnPPeXUJ3pequ6XUk6F7+JvzQBAANnjOqIhU5epsNhSr9jG+nO3lqZHAk6JIAIAAcJdVKwhU9O0L9etSxuF65U7KafC9xFEACBAjPl8rZanH1REtTBNvjdeNSmnwg8QRAAgAEz7eYdm/JJRVk5tXo9yKvwDQQQA/Fzq9gMa8/laSdKTN16qbpRT4UcIIgDgx/a4jujRaZ5yau92TfTotZRT4V8IIgDgp8qXU1s1Dte/7mhPORV+hyACAH7Isiw9/5mnnBpZvYomJVJOhX8iiACAH5r2c7pm/pqhEMqp8HMEEQDwM6nbD2jsF0fLqa107SUNDE8EnDuCCAD4kaycY3dO7d2uiYZce6HpkYDzQhABAD9xtJyanecpp758J+VU+D+CCAD4Acuy9NyctVqR4SmnTk5MUI2qlFPh/wgiAOAHpv6crlmpnnLq+AGd1KxeDdMjAZWCIAIAPu7X7Qc0tvTOqaNuaqVrKKcigBBEAMCH7c45rEenpqmoxNKf2jfRI9dQTkVgIYgAgI86UlisIVPSlJ1XwJ1TEbAIIgDggyzL0t/mrNHKnTmUUxHQCCIA4IOmLN2hj9J2KsQhvXUP5VQELoIIAPiYn7fu17gv1kmSnu7VSldfTDkVgYsgAgA+ZHfOYQ2dvkxFJZZu7dBUD19NORWBjSACAD6ifDm1dZMI/fN2yqkIfAQRAPABlmXpr6Xl1No1qmhyYryqVw01PRZgO4IIAPiAD/53hz4+Wk4dEKfoupRTERwIIgBg2M9b9+vv//WUU0f3aq2rLq5veCLAewgiAGBQ5sHD+vM0Tzm1T8emeujqGNMjAV5FEAEAQ44UFmvI1DTtzy9QmyYReuk2yqkIPgQRADDAsiw9O3uNVu3MUZ0aVTSJciqCFEEEAAz4z5Lt+mTZ0TunUk5F8CKIAICXLd26X3//cr0k6ZmbW+vKiyinIngRRADAi3YdPKyh05apuMRS345N9eBVlFMR3AgiAOAlR++cerScmkw5FSCIAIA3WJalZz5drdW7KKcC5RFEAMAL3vtpuz5dvkuhIQ69TTkVKEMQAQCbLdmSrRfnesqpo3u1UlfKqUAZgggA2Gjn74c0bPpyFZdY6tfpAsqpQAUEEQCwydE7px7IL1DsBRFKvq0d5VSgAoIIANjgaDl1zS6X6tasqkmJCapWhXIqUBFBBABs8D8VyqkX1K5ueiTAJxFEAKCSLdmSrX+UllOfvbm1urSsZ3giwHcRRACgEpUvp94Wd4EeuLKF6ZEAn0YQAYBKcrigWI9M8ZRT210QqX/0o5wKnA5BBAAqgWVZGv3pKq3NdKlezaqamBhPORU4AwQRAKgE7/64TXNWZCo0xKG3KKcCZ4wgAgDnacnmbCV/tUGS9NfelFOBs2FrEFm8eLFuueUWNW3aVA6HQ3PmzLFzOQDwuowDhzR0+rKycur9XVuYHgnwK7YGkfz8fHXo0EFvvfWWncsAgBFHy6m/HyqknAqcozA7n7xXr17q1auXnUsAgBGWZenpT1dp3W5POXUS5VTgnNARAYBz8O6P2/TZikyFhTj09sA4NaWcCpwTW6+InC232y2321227XK5DE4DACf346Zjd079a+/WuuJCyqnAufKpKyLJycmKjIwse0RHR5seCQCOk3HgkIbNWKYSS7o9Lkr3UU4FzotPBZHRo0crJyen7JGRkWF6JAAoc7igWIOnpOngoUK1j4rUi/1iKacC58mn3ppxOp1yOp2mxwCAE1iWpVGfrNL63S7Vr1VVEwdRTgUqg61BJC8vT5s3by7b3rZtm1asWKG6deuqWbNmdi4NAJXqnR+26ouVpeXUeyinApXF1iCSmpqq7t27l20nJSVJku677z69//77di4NAJXmx03Zeqn0zqnP3dJGnSmnApXG1iDSrVs3WZZl5xIAYKvy5dQ746OUeEVz0yMBAcWnyqoA4EsOFRTp4Q9SdfBQoTpE19bf+1JOBSobQQQATsKyLI36eJU2ZOWWllPjKKcCNiCIAMBJTF68Vf9dtVthIQ5NGBivJpGUUwE7EEQAoIIfNu3TP+d5yqnP39JGl8fUNTwRELgIIgBQTvr+Qxo+Y7lKLOmuhCgNopwK2IogAgClDhUUafCUY+XUcX0opwJ2I4gAgDzl1CfLyqlOTeLOqYBXEEQAQNKkxVv1ZWk5NWVQnBpHVjM9EhAUCCIAgt7i3/bpX0fLqbe21WUtKKcC3kIQARDUduzPLyun3p0QrUGd+TtYgDcRRAAErXx3kQZ/kKacw4XqGF1b4/q2pZwKeBlBBEBQ8pRTV2rjHk85deKgeDnDKKcC3kYQARCUUhZt0dzVWaoS6tBEyqmAMQQRAEFn4ca9evnrjZKk529pqwTKqYAxBBEAQWV7dr4em7FcliUNuDxaAymnAkYRRAAEjXx3kR6ZkibXkSLFNautMbdSTgVMI4gACArly6kNwp1KoZwK+ASCCICgMGHh8eXURhGUUwFfQBABEPC+37hXr3zjKaeOvTVW8c0ppwK+giACIKBtz87XiLJyajPdQzkV8CkEEQABK89dpMFTUsuVU9uYHglABQQRAAHJsiw9+dFK/bYnTw3DuXMq4KsIIgAC0oSFW/TVGk85NWVQvBpSTgV8EkEEQMD5fsOxcuq4PrGKb17H8EQA/ghBBEBA2Zadr8dmesqp93RupgGXU04FfBlBBEDAyHMXafAHqco9UqT45nU05pa2pkcCcBoEEQABwbIsPfHhSm3am6dGEU6lDIxT1TB+xQG+jp9SAAHh7e83a97aLFUNDaGcCvgRgggAv/fdhj16df5vkqRxfdoqrhnlVMBfEEQA+LWt+/I0YsYKWZY0sHMz9aecCvgVgggAv5V7pFCDp6Qp112khOZ19DzlVMDvEEQA+KWSEkt/+XClNpeWUycMopwK+CN+agH4pbe+36xv1u1R1dAQTRwUr4bhlFMBf0QQAeB3vl2/R68v8JRTX+gbq06UUwG/RRAB4Fe27MvTyJmecmriFc1112XRpkcCcB4IIgD8Ru6RQs+dU91FuqxFHf3tT21MjwTgPBFEAPiFkhJLSR+u1JZ9+WocUU0TBsZTTgUCAD/FAPzC+O82a/66PaoaFqKJifFqEO40PRKASkAQAeDzFqw7vpzaMbq22YEAVBqCCACftmVfnh6ftUKSdG+X5rorgXIqEEgIIgB8Vvly6uUt6lJOBQIQQQSATyopsfT4LE85tUlkNb09ME5VQvmVBQQafqoB+KQ3v9ukBetLy6mDKKcCgYogAsDnzF+3R28s2CRJerFvrDpQTgUCFkEEgE/ZvPdYOfW+Ls11J+VUIKARRAD4DNeRQg2ekqq80nLqXymnAgHPK0FkwoQJiomJUbVq1RQfH68ffvjBG8sC8CMlJZYen7lCW0vLqRMGUU4FgoHtP+WzZs3SyJEj9eyzz2r58uW6+uqr1atXL6Wnp9u9NAA/8sa3m/Tthr2qGhaiSYnxql+LcioQDGwPIq+99poefPBBPfTQQ2rdurXeeOMNRUdHKyUlxe6lAfiJr9dm6c1vPeXU5H7t1D6qttmBAHhNmJ1PXlBQoLS0ND399NPH7e/Zs6eWLFlywvFut1tut7ts2+Vy2TLXxqxcTV681ZbnBnD2vl6bJUm6v2sL3R4fZXgaAN5kaxDJzs5WcXGxGjVqdNz+Ro0aKSsr64Tjk5OTNXbsWDtHkiTtcR3RJ8t22r4OgDPXOaaunu3d2vQYALzM1iBylMPhOG7bsqwT9knS6NGjlZSUVLbtcrkUHV35H92LqV9To3u1qvTnBXBuwqtVUZ+OTSmnAkHI1iBSv359hYaGnnD1Y+/evSdcJZEkp9Mpp9P+glp03Rp65NqWtq8DAABOzdb//ahatari4+M1f/784/bPnz9fXbt2tXNpAADgB2x/ayYpKUmJiYlKSEhQly5dNHnyZKWnp2vIkCF2Lw0AAHyc7UHk7rvv1v79+zVu3Djt3r1bsbGxmjt3rpo3b2730gAAwMc5LMuyTA/xR1wulyIjI5WTk6OIiAjT4wAAgDNwNq/fVNQBAIAxBBEAAGAMQQQAABhDEAEAAMYQRAAAgDEEEQAAYAxBBAAAGEMQAQAAxhBEAACAMQQRAABgDEEEAAAYQxABAADGEEQAAIAxBBEAAGAMQQQAABhDEAEAAMYQRAAAgDEEEQAAYAxBBAAAGEMQAQAAxhBEAACAMQQRAABgDEEEAAAYQxABAADGEEQAAIAxBBEAAGAMQQQAABhDEAEAAMYQRAAAgDEEEQAAYAxBBAAAGEMQAQAAxhBEAACAMQQRAABgDEEEAAAYQxABAADGEEQAAIAxBBEAAGAMQQQAABhDEAEAAMYQRAAAgDEEEQAAYAxBBAAAGEMQAQAAxhBEAACAMQQRAABgjK1B5MUXX1TXrl1Vo0YN1a5d286lAACAH7I1iBQUFOjOO+/Uo48+aucyAADAT4XZ+eRjx46VJL3//vt2LgMAAPyUrUHkbLndbrnd7rLtnJwcSZLL5TI1EgAAOEtHX7ctyzrtsT4VRJKTk8uuopQXHR1tYBoAAHA+cnNzFRkZecpjzjqIjBkz5qRhobxff/1VCQkJZ/vUGj16tJKSksq2S0pKdODAAdWrV08Oh+Osny/QuFwuRUdHKyMjQxEREabHCWica+/hXHsX59t7gvlcW5al3NxcNW3a9LTHnnUQGTZsmPr373/KY1q0aHG2TytJcjqdcjqdx+3j0zYnioiICLp/1KZwrr2Hc+1dnG/vCdZzfborIUeddRCpX7++6tevf9YDAQAAVGRrRyQ9PV0HDhxQenq6iouLtWLFCknSRRddpFq1atm5NAAA8AO2BpHnnntO//nPf8q2O3XqJEn6/vvv1a1bNzuXDkhOp1PPP//8CW9fofJxrr2Hc+1dnG/v4VyfGYd1Jp+tAQAAsAF/awYAABhDEAEAAMYQRAAAgDEEEQAAYAxBJAC43W517NhRDoej7CPSqDzbt2/Xgw8+qJiYGFWvXl0tW7bU888/r4KCAtOjBYQJEyYoJiZG1apVU3x8vH744QfTIwWc5ORkXXbZZQoPD1fDhg3Vt29fbdy40fRYQSE5OVkOh0MjR440PYrPIogEgFGjRp3RbXRxbjZs2KCSkhJNmjRJa9eu1euvv66JEyfqmWeeMT2a35s1a5ZGjhypZ599VsuXL9fVV1+tXr16KT093fRoAWXRokUaOnSoli5dqvnz56uoqEg9e/ZUfn6+6dEC2q+//qrJkyerffv2pkfxaXx818999dVXSkpK0ieffKK2bdtq+fLl6tixo+mxAt7LL7+slJQUbd261fQofq1z586Ki4tTSkpK2b7WrVurb9++Sk5ONjhZYNu3b58aNmyoRYsW6ZprrjE9TkDKy8tTXFycJkyYoBdeeEEdO3bUG2+8YXosn8QVET+2Z88ePfzww5oyZYpq1KhhepygkpOTo7p165oew68VFBQoLS1NPXv2PG5/z549tWTJEkNTBYecnBxJ4t+wjYYOHarevXurR48epkfxebbeWRX2sSxL999/v4YMGaKEhARt377d9EhBY8uWLRo/frxeffVV06P4tezsbBUXF6tRo0bH7W/UqJGysrIMTRX4LMtSUlKSrrrqKsXGxpoeJyDNnDlTaWlpSk1NNT2KX+CKiI8ZM2aMHA7HKR+pqakaP368XC6XRo8ebXpkv3Wm57q8zMxM3XTTTbrzzjv10EMPGZo8sDgcjuO2Lcs6YR8qz7Bhw7Rq1SrNmDHD9CgBKSMjQyNGjNC0adNUrVo10+P4BToiPiY7O1vZ2dmnPKZFixbq37+/vvjii+N+YRcXFys0NFQDBw487m/84OTO9Fwf/WWSmZmp7t27q3Pnznr//fcVEkKOPx8FBQWqUaOGPvroI/Xr169s/4gRI7RixQotWrTI4HSBafjw4ZozZ44WL16smJgY0+MEpDlz5qhfv34KDQ0t21dcXCyHw6GQkBC53e7jvgaCiN9KT0+Xy+Uq287MzNSNN96ojz/+WJ07d1ZUVJTB6QLPrl271L17d8XHx2vq1Kn8IqkknTt3Vnx8vCZMmFC2r02bNurTpw9l1UpkWZaGDx+u2bNna+HChbr44otNjxSwcnNztWPHjuP2PfDAA2rVqpWeeuop3g47CToifqpZs2bHbdeqVUuS1LJlS0JIJcvMzFS3bt3UrFkzvfLKK9q3b1/Z1xo3bmxwMv+XlJSkxMREJSQkqEuXLpo8ebLS09M1ZMgQ06MFlKFDh2r69On67LPPFB4eXtbBiYyMVPXq1Q1PF1jCw8NPCBs1a9ZUvXr1CCF/gCACnMY333yjzZs3a/PmzSeEPC4onp+7775b+/fv17hx47R7927FxsZq7ty5at68uenRAsrRj0d369btuP3vvfee7r//fu8PBJTDWzMAAMAY2nYAAMAYgggAADCGIAIAAIwhiAAAAGMIIgAAwBiCCAAAMIYgAgAAjCGIAAAAYwgiAADAGIIIAAAwhiACAACMIYgAAABj/g+jPKqr2RDxnAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(-5.0, 5.0, 0.1)\n",
    "y = relu(x)\n",
    "plt.plot(x, y)\n",
    "plt.ylim(-1.0, 5.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4f948ac",
   "metadata": {},
   "source": [
    "### 3.4 3层神经网络的实现\n",
    "\n",
    "使用Numpy多维数组实现\n",
    "\n",
    "> 神经网络第一层 \n",
    ">> 输入$x_1$, $x_2$，输出三个结果$z_{11}$, $z_{12}$, $z_{13}$\n",
    "\n",
    "> 神经网络第二层 \n",
    ">> 输入$z_{11}$, $z_{12}$, $z_{13}$，输出两个结果$z_{21}$, $z_{22}$\n",
    "\n",
    "> 神经网络第三层 \n",
    ">> 输入$z_{21}$, $z_{22}$，输出两个结果$y_{1}$, $y_{2}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "4efede18",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2,)\n",
      "(2, 3)\n",
      "(3,)\n",
      "A1:  [0.3 0.7 1.1]\n"
     ]
    }
   ],
   "source": [
    "# 定义X,W1,B1\n",
    "X = np.array([1.0, 0.5])\n",
    "W1 = np.array([[0.1, 0.3 ,0.5], [0.2, 0.4, 0.6]])\n",
    "B1 = np.array([0.1, 0.2, 0.3])\n",
    "\n",
    "# 查看X,W1,B1\n",
    "print(X.shape)\n",
    "print(W1.shape)\n",
    "print(B1.shape)\n",
    "\n",
    "# 计算\n",
    "A1 = np.dot(X, W1) + B1\n",
    "print(\"A1: \", A1)\n",
    "\n",
    "# 激活函数\n",
    "Z1 = sigmoid(A1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "17ffa355",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义W2,B2\n",
    "W2 = np.array([[0.1, 0.4], [0.2, 0.5],  [0.3, 0.6]])\n",
    "B2 = np.array([0.1, 0.2])\n",
    "\n",
    "# 计算\n",
    "A2 = np.dot(Z1, W2) + B2\n",
    "\n",
    "# 激活函数\n",
    "Z2 = sigmoid(A2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "01ebcf55",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.31682708, 0.69627909])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 定义恒等函数\n",
    "\n",
    "def identity_function(x):\n",
    "    return x\n",
    "\n",
    "# 定义W3,B3\n",
    "W3 = np.array([[0.1, 0.3], [0.2, 0.4]])\n",
    "B3 = np.array([0.1, 0.2])\n",
    "\n",
    "# 计算\n",
    "A3 = np.dot(Z2, W3) + B3\n",
    "\n",
    "# 最后得到结果，没用激活函数\n",
    "Y = identity_function(A3)\n",
    "Y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9521edb",
   "metadata": {},
   "source": [
    "### softmax函数\n",
    "\n",
    "softmax函数一般用于分类问题\n",
    "\n",
    "$$\n",
    "y_k = \\frac {exp(a_k)}{\\sum_{i=1}^n exp(a_i)}\n",
    "$$\n",
    "\n",
    "由上式可知，整体的和$\\sum_{k=1}^n y_k$为1。\n",
    "\n",
    "但是当$a_k$非常大时，会导致分子分母都是无穷大inf，相除结果不确定。\n",
    "所以我们都乘上一个c，防止溢出。这个c一般是输入信号最大项。\n",
    "\n",
    "$$\n",
    "\\begin{equation}\\begin{split} \n",
    "y_k = \\frac {exp(a_k)}{\\sum_{i=1}^n exp(a_i)} &= \\frac {Cexp(a_k)}{C\\sum_{i=1}^n exp(a_i)} \\\\[2ex]\n",
    "&= \\frac {exp(a_k + \\log C)}{\\sum_{i=1}^n exp(a_i + \\log C)} \\\\[2ex]\n",
    "&= \\frac {exp(a_k + C')}{\\sum_{i=1}^n exp(a_i + C')}\n",
    "\\end{split}\\end{equation}\n",
    "$$\n",
    "\n",
    "输出的每个值都在(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "3206cfaf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1010\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([9.99954600e-01, 4.53978686e-05, 2.06106005e-09])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def softmax(x):\n",
    "    c = np.max(x) \n",
    "    exp_x = np.exp(x-c)\n",
    "    sum_exp_x = np.sum(exp_x)\n",
    "    y = exp_x/sum_exp_x\n",
    "\n",
    "    return y\n",
    "\n",
    "softmax(np.array([1010,1000,990]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
